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INTRODUCTION 


In  breast  imaging,  it  has  been  well  recognized  that  merging  information  from  different 
images  will  greatly  benefit  the  diagnosis  of  breast  cancer  [1-4],  as  well  as  contribute  to 
the  assessment  of  tumor  response  and  image-guided  therapy.  However,  interpreting 
images  from  different  modalities  is  not  trivial  as  different  images  of  the  same  lesion  may 
exhibit  different  physical  lesion  characteristics,  and  the  image  acquisitions  are  performed 
under  different  breast  positioning  protocols.  Also,  the  breast  is  a  non-rigid  object,  and 
thus  conventional  image  registration  methods  are  not  appropriate.  So  the  essential 
problem  of  merging  information  from  different  modalities  is  to  address  the  non-trivial 
task  of  identifying  corresponding  images  of  lesions  as  seen  with  different  views  and/or 
different  imaging  modality  techniques.  The  purpose  of  this  research  is  to  develop 
correlative  feature  analysis  methods  for  integrating  image  information  from  multi¬ 
modality  breast  images,  taking  advantage  of  the  information  from  different  views  and/or 
different  modalities,  and  thus  improving  the  sensitivity  and  specificity  of  breast  cancer 
diagnosis.  A  novel  aspect  of  the  proposed  research  is  the  development  of  correlative 
feature  analysis  (CFA)  into  the  decision-making  process.  Our  hypothesis  is  that  the 
proposed  correlative  feature  analysis  can  benefit  computerized  corresponding  image 
analysis,  leading  to  improved  discrimination  between  corresponding  and  non¬ 
corresponding  lesion  pairs.  This  report  summarizes  the  progress  of  this  Predoctoral 
Traineeship  Award  project  made  by  the  recipient  during  the  past  three  years. 
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BODY 


Training  Accomplishments 

At  the  time  of  this  report,  the  recipient  of  the  Predoctoral  Traineeship  Award  has  taken  22 
out  of  the  22  required  courses  towards  the  Ph.D.  degree  in  medical  physics.  The  courses 
include  physics  of  medical  imaging,  physics  of  radiation  therapy,  mathematics  for 
medical  physicists,  image  processing,  statistics,  machine  learning,  numerical  computation, 
computer  vision,  anatomy  of  the  body,  radiation  biology,  and  teaching  assistant  training. 

Research  Accomplishments 

1,  Database  collection 

We  have  collected  a  multi-modality  breast  image  database  from  the  University  of 
Chicago  Medical  Center  (UCMC)  under  protocols  approved  by  the  Institute  Board 
Review  (IRB).  This  database  includes  full-field  digital  mammographic  (FFDM)  images, 
breast  ultrasound  (US)  images  and  breast  dynamic  contrast-enhanced  magnetic  resonance 
(DCE-MR)  images.  The  FFDM  database  consists  of  229  malignant  and  164  benign 
lesions.  All  the  images  were  obtained  from  GE  Senographe  2000D  systems  with  a  spatial 
resolution  of  lOOpmxlOOpm.  The  US  database  consists  of  205  malignant  solid  lesions, 
113  simple  cysts  and  139  benign  solid  lesions.  The  US  images  were  obtained  with  a 
Philips  HDI  5000  US  unit  and  a  12-5MHz  linear  array  probe.  The  pixel  size  varied  from 
53  pm  to  212  pm,  with  the  average  value  of  1 14  pm.  The  MR  database  consists  of  359 
malignant  and  133  benign  lesions.  In  MR  imaging,  patients  were  scanned  in  the  prone 
position  with  a  standard  double  breast  coil  on  a  1.5  T  whole-body  GE  MRI  system. 
Images  were  obtained  by  the  use  of  a  T1 -weighted  3D  spoiled  gradient  echo  sequences. 
For  each  case,  one  pre-contrast  and  five  post-contrast  series  were  taken.  Each  series 
contained  60  coronal  slices  with  an  image  of  128x256  pixels  and  a  range  of  planar  spatial 
resolution  from  1.25x1. 25mm  to  1.6x1. 6mm  .  Slice  thickness  ranged  from  3  to  4  mm 
depending  on  breast  size. 

All  the  lesions  in  the  multi-modality  database  were  identified  by  expert  breast 
radiologists  based  on  visual  criterion  and  either  biopsy  or  aspiration  proven  reports.  An 
expert  radiologist  (Dr.  C.  S.)  also  helped  us  identify  the  correspondence  of  lesions 
appeared  in  different  modality  images. 

2,  Investigation  of  lesion  segmentation  on  FFDM  images 
2.1  Development  of  a  dual-stage  segmentation  algorithm 

Mass  lesion  segmentation  on  mammograms  is  a  challenging  task  since  mass  lesions  are 
usually  embedded  and  hidden  in  varying  densities  of  parenchymal  tissue  structures.  We 
have  developed  a  dual-stage  method  for  automatic  delineation  of  lesion  boundaries  on 
FFDM  images  [5].  This  method  utilizes  a  geometric  active  contour  model  that  minimizes 
an  energy  function  based  on  the  homogeneities  inside  and  outside  of  the  evolving  contour. 
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Prior  to  the  application  of  the  active  contour  model,  a  radial  gradient  index  (RGI)  based 
segmentation  method  is  applied  to  yield  an  initial  contour  closer  to  the  lesion  boundary 
location  in  a  computationally  efficient  manner.  Based  on  the  initial  segmentation,  an 
automatic  background  estimation  method  is  applied  to  identify  the  effective  circumstance 
of  lesion,  and  a  dynamic  stopping  criterion  is  implemented  to  terminate  the  contour 
evolution  when  it  reaches  the  lesion  boundary.  A  full  description  of  the  method  is  in 
reference  [5]  which  is  attached  as  Appendix  A. 

2.2  Evaluation  of  the  proposed  segmentation  algorithm 

By  using  the  FFDM  database  described  above,  we  compared,  both  qualitatively  and 
quantitatively,  the  proposed  algorithm  with  a  conventional  region-growing  method  [6] 
and  a  RGI-based  algorithm  [7].  Figure  1  shows  an  example  of  lesion  segmentation  using 
region  growing,  RGI-based  segmentation  alone  and  the  proposed  dual-stage  segmentation 
method.  The  result  of  the  dual-stage  segmentation  method  visually  demonstrates  a  closer 
agreement  with  the  radiologist’s  outline  of  the  lesion. 


(a) 

Figure  1 :  Segmentation  results  for  a  malignant  lesion,  (a)  radiologist’s  outline,  (b)  region 
growing,  (c)  RGI-based  segmentation  and  (d)  the  dual-stage  segmentation  method. 


We  quantitatively  evaluated  the  performance  difference  among  the  three  automatic 
segmentation  methods  by  use  of  the  area  overlap  ratio  between  computer  segmentation 
and  manual  segmentation  by  an  expert  radiologist.  Figure  2  shows  the  fraction  of  lesions 
correctly  segmented  at  various  overlap  threshold  levels.  At  the  overlap  threshold  of  0.4, 
for  benign  lesions,  87%  of  the  images  are  correctly  segmented  with  the  proposed  method, 
while  72%  and  81%  of  the  images  are  correctly  segmented  by  the  region-growing  and 
RGI-based  methods,  respectively.  For  malignant  lesions,  84%  of  the  images  are  correctly 
segmented  with  the  proposed  method,  while  66%  and  67%  of  the  images  are  correctly 
segmented  by  region-growing  and  RGI-based  methods,  respectively. 
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Area  overlap  ratio  threshold 


Figure  2:  Performance  of  three  different  segmentation  methods  in  terms  of  area  overlap 
ratio  on  a  clinical  FFDM  database.  Left:  Evaluated  on  327  benign  images;  Right: 
Evaluated  on  412  malignant  images.  The  results  show  that  the  dual-stage  segmentation 
method  is  statistically  superior  to  both  region-growing  and  RGI -based  method. 


2.3  Investigation  of  lesion  segmentation  to  FFDM  computer-aided  diagnosis  tCAD) 


We  continued  to  evaluate  the  performance  of  the  proposed  segmentation  algorithm  for 
the  task  of  assessing  the  likelihood  of  malignancy  of  lesion.  As  characteristic  features  are 
extracted  from  the  segmented  lesion  itself,  or  the  neighborhood  of  the  segmented  lesion, 
the  more  accurately  segmented  lesions,  the  more  meaningful  and  stable  features  are 
expected.  In  our  study  [8],  15  features  [9]  were  automatically  extracted  to  quantify  the 
characteristics  of  spiculation,  margin,  contrast,  shape  and  texture.  An  effective  subset  of 
features  were  automatically  selected  by  a  stepwise  method  and  merged  with  a  Bayesian 
Artificial  Neural  Network  (BANN)  [10]  to  yield  a  discriminant  score,  which  estimates 
the  probability  of  malignancy  (PM)  for  a  given  lesion.  The  performance  of  individual 
features  and  the  selected  feature  subset  was  evaluated  using  receiver  operating 
characteristic  (ROC)  analysis  [11],  with  the  area  under  the  ROC  curve  (AUC)  as  a  figure 
of  merit. 


We  compared  the  classification  performance  of  the  proposed  method  with  that  of  a 
conventional  region-growing  method,  which  was  used  for  our  previously  developed  CAD 
system  for  mammographic  images.  The  FFDM  dataset  used  in  this  study  included  146 
malignant  and  134  benign  cases.  The  results  showed  that  the  performances  of  most  of  the 
spiculation  features  were  improved  with  the  dual-stage  segmentation  method.  However, 
we  failed  to  observe  improvement  for  other  features.  In  leave-one-out  evaluation  by 
lesion,  the  effective  feature  subset  by  the  dual-stage  segmentation,  including  two 
spiculation  features  and  one  gradient  texture,  yielded  an  AUC  of  0.78,  while  the  prior- 
selected  feature  subset  from  the  region  growing  segmentation,  including  three  spiculation 
features,  one  margin  sharpness  and  one  average  gray  level  of  lesion,  yielded  an  AUC  of 
0.72.  The  difference  is  statistically  significant  (p=0.04).  This  work  has  been  reported  at 
the  AAPM  annual  meeting,  2008.  A  more  detailed  summary  is  in  reference  [8],  which  is 
also  attached  as  Appendix  B. 
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3.  Investigation  of  correlative  feature  analysis  across  (CFA)  mammographic  views 

We  firstly  investigated  on  a  dual-stage  correlative  feature  analysis  method  to  address  the 
task  of  classifying  corresponding  images  of  lesions  as  seen  in  different  mammographic 
views  [12],  The  proposed  CFA  method  is  different  from  conventional  image  registration 
methods  in  the  following  two  aspects.  (1)  The  task  of  image  registration  is  to  align  two 
images  known  to  represent  the  same  object,  while  CFA  is  to  assess  the  probability  that 
the  given  two  images  represent  the  same  object.  (2)  The  key  point  of  image  registration  is 
to  determine  a  geometrical  transformation  that  minimizes  some  cost  functions  defined  by 
intensities,  contours  and  mutual  information,  in  which  various  geometrical  landmarks, 
such  as  control  points  and  inherent  image  landmarks  are  identified  and  matched.  The 
CFA  technique  is  feature  based,  which  is  motivated  by  the  studies  on  fusion  of  two-view 
information  for  computer-aided  detection,  as  well  as  our  prior  research  on  the  task  of 
automated  classification  of  breast  lesions,  i.e.,  in  the  determination  of  benign  and 
malignant  breast  lesions  based  on  computer-extracted  features.  A  full  description  of  the 
method  is  in  reference  [12],  which  is  also  attached  as  Appendix  C. 

3.1  Design  of  corresponding  and  non-corresponding  datasets 

Based  on  the  correspondence  of  lesions  identified  by  the  radiologist,  we  constructed  123 
corresponding  pairs  and  82  non-corresponding  pairs  [12].  Each  pair  consists  of  a  cranio- 
caudal  (CC)  view  and  a  medio-lateral  (ML)  view.  Considering  the  most  realistic  scenario 
of  lesion  mismatch  in  clinical  practice,  the  non-corresponding  pairs  were  constructed 
from  cases  of  the  same  patients  but  different  physical  lesions.  In  Figure  3,  (a)  shows  an 
example  with  multiple  lesions  seen  on  mammograms  in  CC  and  ML  views,  and  (b)  lists 
the  corresponding  and  non-corresponding  pairs  constructed  from  (a). 


Corresponding 

Non-Corresponding 

A(CC)-A(ML) 

A(CC)-B(ML) 

B(CC)  -  B(ML) 

B(CC)-A(ML) 

(b) 


Figure  3:  (a)  An  example  of  two  lesions  in  the  same  breast  as  seen  in  CC  view  (left)  and 
ML  view  (right).  The  arrow  indicates  the  correspondence  of  the  same  physical  lesion  in 
different  views,  (b)  The  corresponding  and  non-corresponding  pairs  constructed  from  (a) 
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3.2  Investigation  of  feature  correlation  criterion 


We  evaluated  the  correlation  performance  of  individual  computerized  features  extracted 
from  the  FFDM  images  of  a  lesion  obtained  in  CC  and  ML  views.  In  order  to  evaluate  the 
robustness  of  the  correlation  performance  to  lesion  segmentation,  besides  the 
radiologist’s  outlines,  three  automatic  segmentation  methods  were  employed  to  extract 
the  mass  lesion  from  the  surrounding  tissues,  which  includes  a  conventional  region¬ 
growing  method,  a  RGI-based  method  and  the  newly-developed  dual-stage  segmentation 
method.  15  computer-extracted  features,  which  have  been  described  in  Section  2.3,  were 
calculated  from  each  lesion  in  both  views.  For  each  feature,  Pearson  correlation 
coefficient  between  the  two  views  and  the  p-value  of  the  derived  correlation  coefficient 
were  obtained.  Our  results  show  that  the  features  characterizing  shape,  contrast  and 
texture  performed  better  among  the  15  individual  features  despite  of  segmentation 
methods  and  pathology.  This  is  because  the  features  representing  large-scale  information 
are  less  sensitive  to  the  change  of  position  than  those  representing  small-scale 
information,  which  results  in  the  higher  correlation  between  large-scale  features  from 
different  views  than  that  of  small-scale  features.  This  work  provides  a  guide  for 
discriminating  corresponding  and  non- corresponding  lesion  pairs  within  the  CAD 
framework.  It  is  also  helpful  for  guiding  the  development  of  new  features  to  improve  the 
accuracy  of  image  matching  in  disease  diagnosis  and  prognosis.  A  more  detailed 
summary  can  be  found  in  reference  [13]  which  is  also  attached  as  Appendix  D. 

Mutual  information  (MI)  is  another  measure  of  the  dependence  between  two  variables.  It 
is  well  understood  that  mutual  information  measures  the  general  dependence,  while  the 
correlation  coefficient  measures  the  linear  dependence.  So  we  also  investigated  the 
mutual  information  among  the  features  and  assessed  its  effect  on  the  choice  of 
discriminating  features  as  compared  with  the  use  of  linear  correlation  coefficient  between 
features.  For  each  feature  described  above,  mutual  information  between  the  two  views 
was  obtained  using  a  density  estimation  method  (e.g.,  Parzen  windows)  [14].  However, 
the  dependence  rank  of  features  determined  by  mutual  information  highly  agreed  with 
that  determined  by  linear  correlation  coefficient,  yielding  a  correlation  coefficient  of  0.87. 
This  result  indicated  that  linear  correlation  coefficient  is  a  good  metric  to  represent  the 
dependence  between  features  from  different  views.  Moreover,  since  linear  correlation 
coefficient  is  bounded  to  [-1,  1],  we  will  use  linear  correlation  coefficient  as  the  metric  to 
choose  the  discriminating  features  in  correlative  feature  analysis  across  mammographic 
views. 

3.3  Development  of  new  features 

Since  features  characterizing  large-scale  information  usually  have  better  correlation 
performance,  we  developed  two  sets  of  “large-scale”  features.  Firstly,  we  extracted  a  set 
of  texture  features  based  on  a  gray-level  co-occurrence  matrix  (GLCM).  For  each  region, 
four  GLCMs  were  constructed  along  four  different  directions  of  0°,  45°,  90°  and  135°. 
Assuming  that  there  is  no  directional  texture  features  in  mammograms,  a  non-directional 
GLCM  was  obtained  by  summing  all  the  directional  GLCMs.  Texture  features  were  then 
computed  from  each  non-directional  GLCM.  To  avoid  sparse  GLCMs  for  smaller  lesions, 
the  gray  level  range  of  image  was  scaled  down  to  6  bits,  resulting  in  GLCM  of  size 
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64x64.  Among  the  texture  features,  correlation  feature  performed  best  with  a  correlation 
coefficient  of  0.67  (p-value  <  10"^)  [12]. 

In  clinic  practice,  radiologists  commonly  use  the  distance  from  nipple  to  the  center  of  a 
lesion  to  correlate  the  lesion  in  different  views.  It  is  generally  believed  that  this  distance 
keeps  fairly  constant.  Thus,  we  developed  a  distance  feature  to  measure  the  Euclidean 
distance  between  the  nipple  location  and  the  mass  center  of  lesion.  We  also  developed  an 
automatic  nipple  localization  scheme  to  tracking  nipple  markers  on  each  FFDM  images. 
With  computer-identified  nipples,  the  distance  features  in  CC  views  are  highly  correlated 
with  those  in  ML  views,  yielding  a  correlation  coefficient  of  0.88  (p-value  <  10'^)  [12]. 

3.4  Investigation  of  scheme  to  combine  multiple  classifiers  in  correlative  feature  analysis 

For  each  pairwise  set  of  features  extracted  from  lesions  in  CC  and  ML  views,  a  BANN 
classifier  was  employed  to  merge  each  feature  pair  into  a  correspondence  score,  which  is 
the  estimate  of  the  probability  that  the  two  lesion  images  depict  the  same  physical  lesion. 
As  an  example.  Figure  4(a)  shows  the  distribution  of  distance  feature  generated  from 
lesions  taken  in  different  views  for  corresponding  and  non-corresponding  datasets.  The 
histogram  in  Figure  4(b)  demonstrates,  for  the  corresponding  and  non-corresponding 
datasets,  the  distribution  of  these  correspondence  scores  output  from  the  BANN. 


■  corresponding  pairs 
□  non-corresponding  pairs 

1 

^  ■  — ■  ■  Ji  B  1 

0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1 

Probability  of  corresponding 


Figure  4:  (a)  The  scatter  plot  of  distance  feature  generated  from  lesions  seen  on  CC  and 
ML  views,  (b)  The  distribution  of  the  output  correspondence  scores  of  the  distance 
feature  from  a  BANN  classifier. 


Linear  stepwise  feature  selection  [15]  with  Wilks  lambda  criterion  was  employed  on  all 
feature-based  correspondence  scores  to  select  a  subset  of  scores  for  the  final  task  of 
distinguishing  corresponding  and  non-corresponding  pairs.  Another  BANN  was  then 
trained  with  the  selected  correspondence  scores  to  yield  an  overall  estimate  of  probability 
of  correspondence. 

With  leave-one-out  (by  physical  lesion)  validation,  the  distance  feature  performed  best 
among  all  the  individual  features,  yielding  an  AUC  of  0.81  [12].  A  subset  of  features, 
which  included  distance,  ROI-based  correlation  and  gradient  texture,  was  selected  and 
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merged  with  BANN  to  yield  an  AUC  of  0.87.  The  improvement  by  using  multiple 
features  was  statistically  significant  compared  to  that  of  single  feature  (p  =  0.01). 

3.5  Investigation  of  robustness  of  the  proposed  correlative  feature  analysis 

To  evaluate  the  robustness  of  the  proposed  correlative  feature  analysis  over  different 
mammographic  view  pairs,  we  extended  the  application  of  CFA  to  CC  versus  medio- 
lateral  oblique  (MLO)  views  [16].  We  constructed  104  corresponding  image  pairs  and  95 
non-corresponding  pairs.  The  distance  was  also  the  best  individual  feature  with  AUC  of 
0.78±0.03.  The  selected  feature  subset,  including  distance,  ROI-based  energy  and  ROI- 
based  homogeneity,  yielded  an  AUC  of  0.88±0.02.  This  improvement  was  statistically 
significant  (p  <  0.001).  Although  most  of  the  selected  mathematical  descriptors  were 
different  from  those  based  on  CC  versus  ML  views,  they  did  represent  the  same  physical 
characteristic,  i.e.  texture.  From  the  entire  FFDM  database,  we  constructed  a  dataset  of 
lesions  with  CC,  MLO  and  ML  views,  and  obtained  83  corresponding  pairs  and  66  non¬ 
corresponding  pairs.  The  leave-one-out  (by  physical  lesion)  was  performed  on  the  paired 
lesions  only.  The  proposed  CFA  method  yielded  an  AUC  of  0.87±0.02  for  CC  versus  ML 
views  and  0.90±0.02  for  CC  versus  MLO  views.  The  difference  was  not  statistically 
significant  (p=0.49).  The  results  showed  that  CFA  was  robust  across  two  different  view 
pairs  (CC  versus  ML  and  CC  versus  MLO).  This  work  has  been  reported  in  the  9^** 
International  Workshop  on  Digital  Mammography  (IWDM)  [16],  please  refer  to 
Appendix  E  for  more  details. 

4,  Investigation  of  correlative  feature  analysis  across  imaging  modalities 

The  computerized  method  developed  in  the  above  work  can  be  extended  for  assessing  the 
probability  of  correspondence  across  different  imaging  modalities.  Here  our  investigation 
will  be  based  on  FFDM  images  versus  DCE-MR  images.  The  key  difference  between  this 
scheme  and  the  prior  mammographic  one  is  that,  in  DCE-MR  images,  all  computerized 
features  will  be  extracted  in  three-dimensional  (3D)  space.  Kinetic,  morphologic,  and 
texture  features  are  widely  used  in  the  computerized  analysis  of  breast  DCE-MR  images 
[17,  18].  However,  in  the  proposed  correlative  feature  analysis  study,  only  morphologic 
and  texture  features  will  be  used,  because  FFDM  images  do  not  carry  functional 
information,  such  as  enhancement-variance  dynamics. 

4.1  Design  of  corresponding  and  non-corresponding  datasets 

Depending  on  which  mammographic  view  was  used,  we  constructed  three  data  sub¬ 
categories  from  the  multi-modality  database:  MR  images  versus  CC  views,  MR  images 
versus  ML  views,  and  MR  images  versus  MLO  views.  Note  that  a  mammographic  image 
does  not  correlate  with  a  single  MR  image,  but  a  set  of  MR  images  depicting  a  physical 
lesion  in  3D  space.  Table  1  lists  the  detailed  information  regarding  the  corresponding  and 
non-corresponding  datasets  in  each  sub-category. 
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Table  1:  The  number  of  lesion  pairs  in  corresponding  and  non-corresponding  datasets. 
The  non-corresponding  pairs  were  constructed  from  cases  of  the  same  breasts  but 
different  physical  lesions. _ _ 


Corresponding  dataset 

Non-corresponding  dataset 

MR  vs.  CC  view 

243 

98 

MR  vs.  ML  view 

219 

88 

MR  vs.  MLO  view 

122 

77 

4.2  Investigation  of  feature  correlation  criterion 

In  the  correlative  feature  analysis  across  mammographic  views,  we  used  Pearson 
correlation  coefficient  as  the  criterion  to  evaluate  the  correlative  performance  of 
computerized  features  from  different  views.  Pearson  correlation  coefficient  is  a  widely 
used  criterion  that  measures  the  linear  correlation  between  two  feature  vectors.  However, 
it  is  more  natural  to  extract  a  general  correlation  between  two  feature  vectors  when  we 
compare  images  from  different  modalities.  Thus,  we  introduced  Spearman  rank  order 
correlation  coefficient  [19]  in  this  study. 

As  compared  to  Pearson  correlation  coefficient,  Spearman  rank  order  correlation 
coefficient  is  a  non-parametric  measure  of  correlation,  that  is,  it  assesses  how  well  an 
arbitrary  monotonic  function  could  describe  the  relationship  between  two  variables, 
without  make  any  other  assumptions  about  the  particular  nature  of  the  relationship 
between  the  variables.  Thus,  Spearman  rank  order  correlation  coefficient  could  measure 
the  non-linear  relationship  between  feature  vectors.  Although  mutual  information  (MI) 
can  also  depict  the  non-linear  relationship  between  variables,  its  calculation,  however,  is 
not  standardized  and  thus  hard  to  infer  statistically.  Moreover,  the  MI  value  is  not 
bounded.  Therefore,  we  used  Spearman  rank  order  correlation  coefficient  as  the  metric  to 
choose  the  discriminating  features  in  correlative  feature  analysis  across  different  imaging 
modalities. 

We  used  lesion  size  feature  to  demonstrate  the  effectiveness  of  Spearman  rank  order 
correlation.  The  lesion  size  feature  in  mammograms  {Fmg)  is  conventionally  calculated  as 
the  equivalent  diameter  that  measures  the  diameter  of  a  circle  yielding  the  same  area  as 
the  segmented  lesion,  while  the  lesion  size  feature  in  MR  images  {Fmr)  is  conventionally 
calculated  as  the  volume  of  the  segmented  lesion.  The  equivalent  diameter  of  lesion  in 

MR  images  can  be  obtained  as:  .  In  Figure  5,  (a)  shows  the  scatter  plot 

between  Fmg  and  Fmr,  while  (b)  shows  the  scatter  plot  between  Fmg  and  F’mr  ■  Table  2 
lists  the  associated  Pearson  correlation  coefficient  (y)  and  Spearman  rank  order 
correlation  coefficient  (p).  The  results  show  that  Spearman  rank  order  correlation 
coefficient  could  effectively  extract  the  non-linear  relationship  between  features  obtained 
from  MG  images  and  MR  images. 
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Figure  5:  (a)  Scatter  plot  of  lesion  equivalent  diameter  in  MG  images  {Fmg)  versus  lesion 
volume  in  MR  images  (b)  Scatter  plot  of  lesion  equivalent  diameter  in  MG  images 

{Fmg)  versus  lesion  equivalent  diameter  in  MR  images  {F’mi^- 


Table  2:  Pearson  correlation  coefficient  (y)  versus  Spearman  rank  order  correlation 
coefficient  (p). 


Fmg  versus  Fmr 

Fmg  versus  F’mk 

Y 

P 

Y 

P 

Corresponding  dataset 

0.49 

0.65 

0.65 

0.65 

Non-corresponding  dataset 

-0.11 

0.03 

-0.06 

0.03 

4.3  Investigation  of  geometric  features 

Investigation  of  geometric  features  is  motivated  by  the  observation  that  radiologists 
ordinarily  use  location  information  to  identify  lesions  in  different  images.  Also,  in  the 
previous  study  of  correlative  feature  analysis  across  mammographic  views,  we  have 
shown  that  the  distance  feature  performs  well  for  the  task  of  finding  the  corresponding 
lesions  in  different  mammographic  views. 

Figure  6  illustrates  the  geometric  configurations  of  CC  view,  MLO  view  and  MR  images. 
Note  that  we  did  not  include  ML  view  in  the  figure  because  its  geometric  property  is 
close  to  that  of  MLO  view.  We  developed  an  automatic  method  to  locate  the  mass  center 
of  a  lesion  L{L^,  Ly,  in  a  Cartesian  coordinate  system.  The  origin  of  the  coordinate 
system  is  set  at  the  nipple,  and  its  z  direction  is  from  nipple  to  the  chest  wall.  Then,  we 
extracted  geometric  feature  pairs  depending  on  which  mammographic  view  was  used,  as 
listed  in  Table  3. 
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Figure  6:  Illustration  of  geometric  configuration  of  CC  view  (a),  MLO  view  (b)  and  MR 
images  (c). 


Table  3:  Geometric  feature  pairs  between  MR  images  and  mammographic  views,  dxz  is 
calculated  as  the  projection  of  the  Euclidean  distance  between  nipple  and  the  center  of 
lesion  on  the  x-z  plane,  and  dyz  is  calculated  as  the  projection  of  the  Euclidean  distance 
between  nipple  and  the  center  of  lesion  on  the  y-z  plane. 


MR  images 

CC  view 

MLO  view 

ML  view 

X 

X 

- 

- 

y 

- 

y 

y 

z 

z 

z 

z 

dxz 

dxz 

- 

- 

dyz 

- 

dyz 

dyz 

For  each  pair-wise  set  of  geometric  features  extracted  from  lesions  in  MR  images  and 
mammographic  views,  a  BANN  classifier  was  employed  to  merge  each  feature  pair  into  a 
correspondence  score,  which  is  the  estimate  of  the  probability  that  the  two  lesion  images 
depict  the  same  physical  lesion.  Table  4  lists  the  AUC  values  of  correspondence  scores 
obtained  from  geometric  feature  pairs,  which  were  evaluated  with  leave-one-out  (by 
lesion)  validation. 


Table  4:  Performance  of  correspondence  scores  obtained  from  geometric  feature  pairs. 


Geometric  features 

MR  VS.  CC 
(AUC  +  se) 

MR  vs.  MLO 
(AUC  +  se) 

MR  vs.  ML 
(AUC  +  se) 

X 

0.59  +  0.04 

- 

- 

y 

- 

0.72  +  0.  04 

0.69  +  0.  03 

z 

0.70  +  0.  03 

0.74  +  0.  04 

0.66  +  0.  04 

dxz 

0.65  +  0.  03 

- 

- 

dyz 

- 

0.75  +  0.  04 

0.73  +  0.  03 
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4.4  Evaluation  of  the  proposed  CFA  method 


Linear  stepwise  feature  selection  with  Wilks  lambda  criterion  was  employed  on  all 
feature-based  correspondence  scores  to  select  a  subset  of  scores  for  the  final  task  of 
distinguishing  corresponding  and  non-corresponding  pairs.  Another  BANN  was  then 
trained  with  the  selected  correspondence  scores  to  yield  an  overall  estimate  of  probability 
of  correspondence. 

For  MR  images  versus  CC  view,  a  subset  of  features,  which  included  three 
morphological  features,  three  texture  features  and  two  geometric  features,  was  selected 
and  merged  with  BANN  to  yield  an  AUC  of  0.78,  with  leave-one-out  (by  physical  lesion) 
validation.  The  improvement  by  using  multiple  features  was  statistically  significant 
compared  to  that  of  single  feature  (p  =  0.02). 

For  MR  images  versus  MLO  view,  a  subset  of  features,  which  included  lesion  size,  two 
texture  features  and  two  geometric  features,  was  selected  and  merged  with  BANN  to 
yield  an  AUC  of  0.86,  with  leave-one-out  (by  physical  lesion)  validation.  The 
improvement  by  using  multiple  features  was  statistically  significant  compared  to  that  of 
single  feature  (p  =<0.00 1). 

For  MR  images  versus  ML  view,  a  subset  of  features,  which  included  two  morphological 
features  and  three  geometric  features,  was  selected  and  merged  with  BANN  to  yield  an 
AUC  of  0.84,  with  leave-one-out  (by  physical  lesion)  validation.  The  improvement  by 
using  multiple  features  was  statistically  significant  compared  to  that  of  single  feature  (p 
=<0.001). 

5.  Multi-modality  breast  cancer  classiRcation  with  mammography  and  DCE-MRI 

In  order  to  evaluate  the  roles  of  corresponding  lesions  and  their  features  in  breast  CAD, 
we  investigated  the  multimodality  breast  cancer  classification  with  mammography  [9] 
and  DCE-MR  images  [17].  In  this  pilot  study  [20],  we  used  a  FFDM  database  including 
321  lesions  (167  malignant  and  154  benign)  and  a  DCE-MRI  database  including  181 
lesions  (97  malignant  and  84  benign).  From  these  two  databases,  we  constructed  a 
multimodality  dataset  of  51  lesions  (29  malignant  and  22  benign).  Mammograms  and 
DCE-MR  images  are  available  for  these  lesions. 

For  each  lesion  on  each  modality,  computer  automatically  segmented  the  mass  lesions 
and  extracted  a  set  of  features.  Linear  stepwise  feature  selection  was  firstly  performed  on 
single  modalities,  yielding  one  feature  subset  for  each  modality.  Then,  these  selected 
features  served  as  the  input  to  another  feature  selection  procedure  when  extracting  useful 
information  from  both  modalities.  The  selected  features  were  merged  by  a  Linear 
Discrinimant  Analysis  (LDA)  into  a  discriminant  score.  ROC  analysis  was  used  to 
evaluate  the  performance  of  each  selected  feature  subset  in  the  task  of  distinguishing 
between  malignant  and  benign  lesions. 

With  leave-one-out  (by  physical  lesion)  evaluation  on  the  multimodality  dataset,  the 
mammography-only  features  yielded  an  AUC  of  0.62  and  the  DCE-MRI-only  features 
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yielded  an  AUC  of  0.90.  The  combination  of  these  two  modalities,  which  included  a 
spiculation  feature  from  mammography  and  a  kinetic  feature  from  DCE-MRI,  yielded  an 
AUC  of  0.94.  The  improvement  of  combining  multi-modality  information  was 
statistically  significant  as  compared  to  the  use  of  mammography  only  (p=10‘"^).  However, 
we  failed  to  show  statistically  significant  improvement  with  the  multi-modality  features 
as  compared  to  DCE-MRI,  mostly  due  to  the  limited  multi-modality  dataset  (p=0.22). 


In  the  previous  studies  [9]  [17],  spiculation  and  kinetic  features  have  been  justified  as  the 
best  features  when  distinguishing  malignant  and  benign  lesions  for  mammography  and 
DCE-MRI,  respectively.  Our  feature  selection  method  correctly  captured  these  two 
features  when  combining  information  from  different  modalities.  The  results  showed  that 
combining  information  from  multiple  modalities  performed  better  than  the  single 
modality  in  the  task  of  distinguishing  between  malignant  and  benign  lesions.  This  work 
has  been  reported  in  SPIE  Medical  Imaging  Conference,  2009  [20],  please  refer  to 
Appendix  F  for  more  details. 
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KEY  RESEARCH  ACCOMPLISHMENTS 


•  Built  a  multi-modality  database,  which  includes  full-field  digital  mammograms, 
breast  ultrasound  images  and  breast  DCE-MR  images. 

•  Developed  a  dual-stage  lesion  segmentation  method  for  FFDM  images,  which 
outperformed  the  performances  of  our  previous  developed  region-growing  method 
and  the  RGI -based  segmentation  method. 

•  Evaluated  the  proposed  dual-stage  segmentation  method  for  the  task  of  assessing  the 
likelihood  of  malignancy  of  a  mass  lesion  on  FFDM  images,  which  yielded  improved 
classification  performance  over  that  with  region-growing  method. 

•  Investigated  feature  correlation  with  both  linear  correlation  coefficient  and  mutual 
information.  The  results  demonstrate  that  the  features  representing  large-scale 
information  of  lesions  usually  have  better  correlation  performance  and  linear 
correlation  coefficient  is  an  appropriate  metric  characterizing  the  dependence 
between  features  from  different  views. 

•  Developed  texture  features  and  distance  feature,  which  have  been  proven  to  be  useful 
in  differentiating  corresponding  and  non-corresponding  image  pairs. 

•  Developed  a  computerized  correlative  feature  analysis  (CFA)  framework  to  identify 
the  correspondence  between  lesions  imaged  in  different  images.  The  two-step 
classification  scheme  not  only  effectively  utilizes  the  information  regarding 
correlation  between  feature  pairs,  but  also  efficiently  combines  multiple  classifiers 
into  a  final  decision. 

•  Evaluated  the  proposed  correlative  feature  analysis  on  two  sets  of  pair-wise 
mammographic  views,  i.e.  CC  versus  ML  and  CC  versus  MLO.  The  results  show  that 
the  proposed  correlative  feature  analysis  is  effective  and  robust  across  different  view 
pairs. 

•  Investigated  feature  correlation  with  both  Pearson  correlation  coefficient  and 
Spearman  rank  order  correlation  coefficient.  The  results  demonstrate  that  Spearman 
rank  order  correlation  coefficient  could  effectively  extract  non-linear  relationship 
between  features  obtained  from  mammographic  images  and  MR  images. 

•  Developed  geometric  features  to  correlate  lesions  in  3D  MR  images  and  lesions  in  2D 
projection  MG  images,  which  have  been  proven  to  be  useful  in  distinguishing 
corresponding  lesions  from  non-corresponding  ones. 

•  Evaluated  the  proposed  correlative  feature  analysis  across  MR  images  and  different 
mammographic  views.  The  results  show  that  the  proposed  method  can  achieve 
statistically  improved  performance  compared  to  the  use  of  single  feature  in  the  task  of 
differentiating  corresponding  and  non-corresponding  lesions. 
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•  Conducted  a  pilot  study  on  computerized  diagnosis  of  breast  lesions  with 
mammography  and  DCE-MRI.  The  results  showed  that  combining  information  from 
multiple  modalities  performed  better  than  the  single  modality  in  the  task  of 
distinguishing  between  malignant  and  benign  lesions. 
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CONCLUSIONS 


The  recipient  of  the  Predoctoral  Traineeship  Award  has  finished  all  the  required  courses 
towards  his  Ph.D.  degree.  These  trainings  have  proven  useful  for  the  recipient  to  achieve 
the  proposed  research  goals. 

During  the  research,  we  have  collected  and  maintained  a  multi-modality  database 
including  full-field  digital  mammograms,  breast  ultrasound  images  and  breast  MR 
images,  which  is  suitable  for  the  proposed  research  on  correlative  feature  analysis  for 
multi-modality  images.  We  have  developed  a  dual-stage  lesion  segmentation  for  FFDM 
images.  Our  results  suggest  that  the  proposed  method  could  improve  the  segmentation 
performance  and  yield  improved  classification  performance  over  that  with  previous- 
developed  region  growing  method. 

In  this  project,  we  developed  a  computerized  method  to  identify  the  corresponding 
lesions  across  mammographic  views.  The  investigation  of  feature  correlation  showed  that 
linear  correlation  coefficient  is  an  appropriate  metric  characterizing  the  dependence 
between  features  from  different  views.  We  also  developed  texture  features  and  distance 
feature,  which  have  been  prove  to  be  useful  in  differentiating  corresponding  and  non¬ 
corresponding  image  pairs.  The  proposed  two-step  correlative  feature  analysis  not  only 
effectively  utilizes  the  information  regarding  correlation  between  feature  pairs,  but  also 
efficiently  combines  multiple  classifiers  into  a  final  decision.  The  results  of  a  further 
investigation  with  different  mammographic  views  have  shown  that  the  proposed 
correlative  feature  analysis  is  effective  and  robust. 

In  order  to  extend  the  proposed  correlative  feature  analysis  to  multi-modality  images,  we 
have  compared  Pearson  correlation  coefficient  and  Spearman  rank  order  correlation 
coefficient.  Our  results  demonstrated  that  Spearman  rank  order  correlation  coefficient 
could  effectively  extract  non-linear  relationship  between  features  obtained  from 
mammographic  images  and  MR  images.  By  including  the  newly  developed  geometric 
features,  we  evaluated  the  performance  of  the  proposed  correlative  feature  analysis  across 
MR  images  and  various  mammographic  views.  The  results  showed  that  the  proposed 
method  can  achieve  statistically  improved  performance  compared  to  the  use  of  single 
feature  in  the  task  of  differentiating  corresponding  and  non-corresponding  lesions. 

At  last,  we  conducted  a  pilot  study  on  computerized  diagnosis  of  breast  lesions  with 
mammography  and  DCE-MRI.  The  results  showed  that  combining  information  from 
multiple  modalities  performed  better  than  the  single  modality  in  the  task  of  distinguishing 
between  malignant  and  benign  lesions. 

Overall,  we  have  achieved  the  goals  for  this  research  and  laid  down  a  solid  foundation  for 
recipient’s  continuation  of  breast  cancer  research.  The  future  work  will  focus  on  the 
research  that  evaluates  how  the  proposed  correlative  feature  analysis  will  facilitate 
radiologists  for  the  task  of  identifying  corresponding  lesions  in  different  images. 
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Appendix  A 

A  dual-stage  method  for  lesion  segmentation  on  digital  mammograms 

Yading  Yuan,®^  Maryellen  L.  Giger,  Hui  Li,  Kenji  Suzuki,  and  Chariene  Sennett 

Department  of  Radiology,  Committee  on  Medical  Physics,  The  University  of  Chicago, 

5841  South  Maryland  Avenue-MC  2026,  Chicago,  Illinois  60637 

(Received  10  June  2007;  revised  4  September  2007;  accepted  for  publication  4  September  2007; 
published  15  October  2007) 

Mass  lesion  segmentation  on  mammograms  is  a  challenging  task  since  mass  lesions  are  usually 
embedded  and  hidden  in  varying  densities  of  parenchymal  tissue  structures.  In  this  article,  we 
present  a  method  for  automatic  delineation  of  lesion  boundaries  on  digital  mammograms.  This 
method  utilizes  a  geometric  active  contour  model  that  minimizes  an  energy  function  based  on  the 
homogeneities  inside  and  outside  of  the  evolving  contour.  Prior  to  the  application  of  the  active 
contour  model,  a  radial  gradient  index  (RGI)-based  segmentation  method  is  applied  to  yield  an 
initial  contour  closer  to  the  lesion  boundary  location  in  a  computationally  efficient  manner.  Based 
on  the  initial  segmentation,  an  automatic  background  estimation  method  is  applied  to  identify  the 
effective  circumstance  of  the  lesion,  and  a  dynamic  stopping  criterion  is  implemented  to  terminate 
the  contour  evolution  when  it  reaches  the  lesion  boundary.  By  using  a  full-held  digital  mammog¬ 
raphy  database  with  739  images,  we  quantitatively  compare  the  proposed  algorithm  with  a  conven¬ 
tional  region-growing  method  and  an  RGTbased  algorithm  by  use  of  the  area  overlap  ratio  between 
computer  segmentation  and  manual  segmentation  by  an  expert  radiologist.  At  an  overlap  threshold 
of  0.4,  85%  of  the  images  are  correctly  segmented  with  the  proposed  method,  while  only  69%  and 
73%  of  the  images  are  correctly  delineated  by  our  previous  developed  region-growing  and  RGI 
methods,  respectively.  This  resulting  improvement  in  segmentation  is  statistically 
signihcant.  ©  2007  American  Association  of  Physicists  in  Medicine.  [DOI:  10.1118/1.2790837] 

Key  words:  Mass  lesion  segmentation,  geometric  active  contour  model,  computer-aided  diagnosis, 
breast  cancer 


I.  INTRODUCTION 

Breast  cancer  is  the  most  common  malignancy  in  American 
women  and  the  second  most  common  cause  of  death  from 
malignancy  in  this  population.  According  to  the  American 
Cancer  Society,  about  178,480  women  in  the  United  States 
will  be  found  to  have  invasive  breast  cancer  in  2007,  and 
about  40,460  women  will  die  from  the  disease  this  year.* 
Although  some  imaging  modalities,  such  as  magnetic  reso¬ 
nance  imaging  (MRI)^’^  and  sonography,^’^  are  currently  be¬ 
ing  investigated  to  improve  sensitivity  and  specihcity  of 
breast  cancer  diagnosis,  x-ray  mammography  is  still  the  most 
prevalent  imaging  procedure  for  the  early  detection  of  breast 
cancer. 

Lesion  segmentation,  which  extracts  the  lesion  from  the 
surrounding  tissues,  is  an  essential  step  in  the  computerized 
analysis  of  mammograms.  As  mass  lesions  are  usually  em¬ 
bedded  and  hidden  in  varying  densities  of  parenchymal 
structures,  the  task  of  lesion  segmentation  is  not  trivial. 
Many  researchers  have  developed  computer  algorithms  for 
this  task.  Huo  et  al.^  employed  a  region-growing  method  to 
hnd  the  contour,  in  which  abrupt  changes  in  size  and  circu- 

j 

larity  were  used  as  the  rules  of  segmentation.  Kupinski  et  al. 
segmented  the  mass  by  applying  either  a  radial  gradient  in¬ 
dex  (RGI)  model  or  a  probabilistic  model  to  the  lesion,  mul- 
tiplied  by  a  constraint  function.  Petrick  et  al.  introduced  a 
segmentation  algorithm  that  combines  a  density-weighted 
contrast  enhancement  hlter  and  a  region  growing  method.  Li 
et  al.‘^  employed  a  multiresolution  Markov  random  held 


model  to  detect  tumors  in  mammographic  images.  Timp  et 
al.^^  employed  both  edge  based  information  as  well  as  a 
priori  knowledge  about  the  gray  level  distribution  of  the  re¬ 
gion  of  interest  (ROI)  around  the  mass,  and  obtained  an  op¬ 
timal  contour  using  dynamic  programming.  To  segment  le¬ 
sions,  Guliato  et  a/.'*  proposed  two  fuzzy  sets  related 
methods — one  employing  a  region  growing  after  fuzzy-sets- 
based  preprocessing,  and  the  other  using  a  fuzzy  region¬ 
growing  method  that  takes  into  account  the  uncertainty 
present  around  the  boundaries  of  tumor.  Li  et  al.  presented 
a  statistical  model  for  enhanced  segmentation  and  extraction 
of  a  suspicious  mass  area  from  mammographic  images.  In 
their  study,  a  morphological  operation  is  derived  to  enhance 
disease  patterns  of  suspected  masses  by  eliminating  unre¬ 
lated  back-ground  clutter,  and  a  model-based  image  segmen¬ 
tation  is  performed  to  localize  the  suspected  mass  areas  using 
stochastic  relaxation  labeling. 

Originally  introduced  by  Kass,  active  contour  models 
(or  snakes)  have  attracted  much  attention  as  image  segmen¬ 
tation  techniques.  An  active  contour  model  minimizes  an  en¬ 
ergy  functional  along  a  deformable  contour,  which  is  inhu- 
enced  by  both  internal  and  external  terms.  The  internal 
energy  controls  the  smoothness  and  elasticity  of  the  contour, 
while  the  external  energy  attracts  the  evolving  contour  to 
deform  toward  salient  image  features,  such  as  edges.  Al¬ 
though  the  active  contour  model  has  been  used  for  segment¬ 
ing  objects  in  a  wide  range  of  medical  applications,  *‘*  *®  to 
the  best  of  our  knowledge,  few  works  have  applied  this 
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model  to  the  task  of  lesion  segmentation  in  mammographic 
images.  Brake  et  al?^  segmented  mass  lesions  by  a  discrete 
active  contour  method,  whose  external  energy  was  deter- 
mined  by  the  image  gradient  magnitude.  Sahiner  et  al.  ap¬ 
plied  an  active  contour  model  that  incorporated  edge  and 
region  analysis,  in  which  the  contour  energy  was  minimized 
by  a  greedy  algorithm.  In  their  work,  however,  the  contour 
was  represented  by  the  vertices  of  an  A^-points  polygon  and 
each  vertex  was  tracked  during  the  process,  which  makes  it 
difficult  for  the  contour  to  adapt  to  a  change  of  topology, 
such  as  splitting  or  merging  parts. 

Differing  from  the  segmentation  methods  mentioned 
above,  in  this  study  we  develop  an  automatic  lesion  segmen¬ 
tation  algorithm  that  employs  a  geometric  active  contour 
model  to  extract  lesions.  Geometric  active  contour 
models  ’  represent  contours  as  a  level  set  of  a  higher- 
dimensional  scalar  function.  The  contours  are  obtained 
only  after  complete  evolution,  thereby  allowing  the  model  to 
handle  the  topological  changes  naturally.  As  mass  lesions 
usually  have  weak  edges,  we  use  a  region-based  active  con- 
tour  model  that  is  based  on  global  image  information,  and 
is  less  sensitive  to  noise  and  the  initial  contour.  In  order  to 
improve  the  computational  efficiency  and  suppress  the  influ¬ 
ence  of  unrelated  structures,  our  previous  RGTbased  seg¬ 
mentation  method^  is  applied  first  to  delineate  an  initial  con¬ 
tour,  which  is  relatively  close  to  the  actual  margin,  and  to 
estimate  the  effective  background.  We  then  exploit  a  dy¬ 
namic  stopping  criterion,  which  is  solely  based  on  the  prop¬ 
erty  of  the  given  image,  to  terminate  the  evolving  procedure 
automatically. 

The  organization  of  this  paper  is  as  follows:  Section  II 
introduces  the  database  used  for  this  study.  Section  III  de¬ 
scribes  the  proposed  segmentation  method.  Section  IV  pre¬ 
sents  the  results,  and  Sections  V  and  VI  give  a  discussion 
and  conclusion,  respectively. 


II.  MATERIALS 

In  this  study,  we  used  a  full-field  digital  mammography 
(FFDM)  database,  which  consists  of  139  benign  (327  mam¬ 
mograms)  and  148  malignant  (412  mammograms)  lesions. 
All  of  the  images  were  collected  from  the  University  of  Chi¬ 
cago  Hospitals  and  obtained  from  GE  Senographe  2000D 
systems  (GE  Medical  Systems,  Milwaukee,  WI)  with  a  spa¬ 
tial  resolution  of  95  /rm  X  95  /rm.  The  masses  were  identi¬ 
fied  and  outlined  by  an  expert  breast  radiologist  based  on 
visual  criterion  and  biopsy-proven  reports.  These  outlines 
were  used  as  the  “gold  standard”  for  calibrating  parameters 
and  evaluating  performance.  The  distributions  of  effective 
projection  diameter,  which  is  defined  as  the  effective  diam¬ 
eter  of  the  area  inside  the  radiologist’s  manually  delineated 
contours,  are  shown  in  Eig.  1. 


III.  METHODS 

The  main  aspects  of  the  proposed  segmentation  method 
include  an  initial  RGI  segmentation,^  background  estimation 


Fig.  1.  Distribution  of  lesions’  effective  diameters  obtained  from  the  FFDM 
database. 


and  trend  correction,  and  an  active  contour  segmentation 
based  on  level  sets.  Eigure  2  shows  the  flow  chart  of  the 
overall  implementation. 


III.A.  Active  contour  model 

The  active  contour  model  relies  on  an  intrinsic  property 
of  image  segmentation:  Eor  an  image  formed  by  two  regions, 
each  segmented  region  should  be  as  homogeneous  as  pos¬ 
sible.  Mathematically,  this  model  can  be  expressed  by  the 
following  energy  function: 


Fig.  2.  Schematic  diagram  of  the  proposed  dual-stage  lesion  segmentation 
algorithm. 
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E{ci,C2,C)  =  fji  ■  length(C) 

+  ^r  \foix,y)  -  cil'^dxdy 

inside(C) 

+  ^2-  I  \foU,y)-C2\'^dxdy,  (1) 

where  fi^Q,  Xj,  k2>Q  are  fixed  weight  parameters,  C  is  the 
evolving  contour  and  length(C)  is  a  regularizing  term  that 
prevents  the  final  contour  from  converging  to  a  small  area 
due  to  noise,  and  Ci  and  C2  are  mean  values  inside  and  out¬ 
side  of  C,  respectively.  Note  that  many  other  active  contour 
models  are  edge-based  as  opposed  to  the  gray-level  based 
method  used  here. 

Equation  (1)  can  be  represented  and  solved  by  level  set 
theory.^®  Level  set  theory,  in  which  the  two-dimensional 
evolving  contour  C  is  represented  implicitly  as  the  zero  level 
set  of  a  three-dimensional  Lipschitz  function  <f>{x,y),  i.e., 
C={{x,y)  e  O;  0(x,y)  =  O},  evolves  the  contour  by  updating 
the  level  set  function  4>{x,y)  at  fixed  coordinates  through 
iterations  instead  of  tracking  the  contour  itself.  The  initial 
level  set  function  4>{x,y)  is  usually  defined  as  the  signed 
distance  function; 


4>{x,y,t  =  0)=  ±d,  (2) 

where  d  is  the  distance  from  (x,y)  to  C(f=0),  where  C{t 
=  0)  corresponds  to  the  initial  contour.  The  plus  (minus)  sign 
is  chosen  if  the  point  (x,y)  is  inside  (outside)  the  initial  con¬ 
tour  C(f=0). 

With  the  evolution  of  the  contour,  the  level  set  function  </> 
cannot  be  held  as  a  signed  distance  function,  nor  can  it  be 
kept  smooth.  In  order  to  maintain  a  smooth  level  set  func¬ 
tion,  and  thus  ensure  numerical  stability  of  evolution,  it  is 
necessary  to  reinitialize  the  evolving  level  set  function  to  a 
signed  distance  function  periodically.  However,  reinitializa¬ 
tion  is  a  computationally  consuming  procedure  as  it  evolves 
solving  the  partial  differential  equation  ^,=  sign(</>,)(l 
-||V^,||),  where  V^,  corresponds  to  the  gradient  of  the  level 
set  function.  In  addition,  most  reinitializing  schemes  tend  to 

27 

move  the  contour  to  some  degree  due  to  numerical  errors. 

A  signed  distance  function  cj),  however,  has  the  intrinsic 
property  that  ||V^||  =  I.  Thus,  it  is  more  natural  to  incorporate 
this  property  into  the  contour  evolution  instead  of  using  the 
independent  reinitializing  procedure  previously  described. 
Thus,  we  can  introduce  another  regularizing  term  in  the 
active  contour  model  in  Eq.  (1): 

£'(ci,C2,C)  =  At  -  length(C)  +  v^\  (1  -  ||V(^,||)^^fxiiy 
+  ^r  \foix,y)  -  cil'^dxdy 

inside(C) 

H-Xj-  I  \foix,y)-C2\^dxdy,  (3) 

outside(C) 


By  replacing  C  with  (l>{x,y)  in  the  energy  functional  in 
Eq.  (3)  and  introducing  the  regularized  versions  of  the 
Heaviside  function  //J^)  =  |[1  H-2/7r arctan((/>/e)]  along 
with  the  corresponding  Dirac  measure  SJy<j))  =  dl d(j)H 
=  e-[7r-  (6^+ </«')]-',  as  given  by  Chen  and  Vese  in,^^  Eq.  (3) 
can  be  expressed  as: 

£'^(ci,C2,</>)  =  At-  I  SJ,^{x,y))\\V (]){x,y)\\dxdy 

Jn 


+  V-  - 
2 


+  Xi 


I  (1 -||V</)(x,y)||)^^fx^/y 
Ja 

■  I  \foix,y)-Ci\'^H^i(f)ix,y))dxdy 
J  n 


-  C2p{l  -  H l(f)ix,y)]}dxdy ,  (4) 


where  the  first  integral  controls  the  length  of  the  contour  and 
the  second  integral  helps  to  smooth  the  level  set  function  and 
thus  avoid  the  need  for  reinitialization. 

By  fixing  Cj  and  C2  and  minimizing  in  terms  of  tp  at 
each  iteration,  the  associated  Euler-Lagrange  equation  can 
be  derived  as: 


8Jy4>)  •  [At  •  -  Xi  •  (/o  -  c{f- H-  X2  •  (/o  -  C2f^ 


+  V  ■  div 


=  0, 


where 


K=  div 


\W4l 


(5) 


(6) 


represents  the  curvature  of  the  contour  C,  and  also  now  in- 
corporates  the  regularizing  term  from  Li  et  al.  This  deriva¬ 
tion,  combining  the  aspect  of  active  contour  without  edges 
and  level  set  without  reinitialization,  is  given  in  the  Appen¬ 
dix  I.  Using  the  gradient  descent  method,  we  can  solve  (p  in 
Eq.  (5)  iteratively  by  letting  ^  be  a  function  of  iteration  t  and 
replace  the  zero  on  the  right-hand  side  of  Eq.  (5)  by  the  time 
derivative  of  <p.  Thus,  we  obtain  a  partial  differential  equa¬ 
tion  as; 


~=^e{4  -Vix  -  K-\y-  {fo-  cif  +\2  -  ifo-  C2f^ 

at 


+  V  •  div 


l|V</>|| 


VcP 


(7) 


The  time  derivative  d(pl  dt  was  approximated  by  a  forward 
finite  difference; 


S(P  (p"^^  -  <p’' 
St  At 


(8) 


where  v  is  a  weighted  parameter  and  fl  represents  the  whole  while  considering  the  numerical  stability  of  the  PDE  solu- 
image  space.  tion,  the  curvature  k  was  approximated  by  a  discretizing 
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scheme  that  combines  both  forward  and  backward  finite  dif¬ 
ferences,  as  suggested  in  Ref.  29. 


:=  A" 


+  A^ 


_ 

((A-I</.;k)2  +  (m(A:^"^.,Al<^”/)' 


(9) 


where 


=  -r 

and  similarly  for 

,  ,  X  (  sng{a)  +  sngib) 

m(a,b)  =  - 


(10) 


(11) 


III.B.  Contour  initialization 


The  energy  function  in  Eq.  (3)  depends  on  the  evolving 
curve  C  in  a  complex  way.  It  is  not  guaranteed  to  be  qua¬ 
dratic  or  even  convex,  and  one  might  find  a  local  minimum 
of  the  energy  function  somewhere  in  the  neighborhood  of  the 
initial  contour.  Thus,  initializing  the  contour  is  a  nontrivial 
task  for  active  contour  models.  Since  lesion  sizes  vary,  it  is 
difficult  to  find  fixed  parameters  (such  as  the  radius  of  a 
circle)  with  which  to  initialize  the  contour  for  an  entire  da¬ 
tabase.  Hence,  we  use  our  previous  RGTbased  segmentation 
method^  to  estimate  the  initial  boundary  of  a  lesion. 

The  RGTbased  segmentation  algorithm^  incorporates 
prior  knowledge  that  mass  lesions  are  roughly  compact,  and 
thus,  the  original  image  f{x,y)  is  multiplied  with  a  two- 
dimensional  constraint  function  G(x,y;/Aj,/r,.,o^)  to  yield  a 
preprocessed  image  h{x,y)  as 

h{x,y)  =f{x,y)  X  Gix,y,fj,^,fiy,a^),  (12) 

where  G(x,y;/u-^, /U,,,,cr^)  is  a  Gaussian  function  centered  at 
the  manually  indicated  seed  point  and  with  variance 

cr^.  The  multiplication  with  the  Gaussian  function  reduces 
the  contribution  of  structures  beyond  the  lesion,  and  thus,  cr 
is  set  to  15  mm  to  accommodate  most  mammographic  lesion 
sizes.  We  have  found  that  the  segmentation  performance  is 
not  strongly  dependent  on  the  choice  of  cr.  Larger  lesions  can 
also  be  segmented  even  though  the  small  deviations  around 
the  margin  of  the  lesion  are  usually  not  delineated  well. 

Starting  from  the  given  seed  point  (yu,j.,yu,,,),  a  series  of 
gray  level  thresholds  are  then  applied  to  the  preprocessed 
image  h{x,y)  to  yield  multiple  contours.  For  each  contour,  an 
RGI  value  is  calculated,  where  RGI  is  defined  as 


RGI(/r^.,/r,,,C,) 


2  I  V/2(x,y)  • 

T,>')gC,-  \ 


r{x,y)  \ 
\r{x,y)\\l 


2  ||V/2(x,y)|| 

(x,_v)  g  C,- 


(13) 


where  C,-  is  the  set  of  points  on  the  /th  contour,  Vfi(x,y)  is 
the  gradient  vector  of  h{x,y)  at  point  (x,y),  and 


Fig.  3.  Illustration  of  defining  the  effective  background.  In  this  figure,  the 
solid  line  represents  the  initial  contour  obtained  by  RGI  segmentation  and 
the  dashdotted  rectangle  is  the  circumscribed  rectangle  of  this  initial  con¬ 
tour.  The  effective  background  is  defined  as  the  region  inside  the  dashed 
rectangle  excluding  the  region  within  the  initial  contour.  An  automatic 
scheme  is  employed  to  determine  the  best  d. 


f(x,y)/||f(x,y)||  is  the  normalized  radial  vector,  the  direction 
of  which  is  calculated  at  position  (x,y)  with  respect  to  the 
seed  point  (yu,j.,yu,,,).  Of  these  contours,  the  one  yielding  the 
maximum  RGI  value  is  chosen  as  the  contour  that  best  de¬ 
lineates  the  lesion  in  the  initial  step. 

RGI  represents  the  average  proportion  of  the  gradients  in 
the  radially  outward  direction.  The  strategy  of  choosing 
maximum  RGI  works  well  for  benign  lesions  as  most  have 
circularlike  shapes  and  smooth  margins.  However,  for  malig¬ 
nant  lesions,  because  of  irregular  shapes  and  spiculate  mar¬ 
gins,  the  resulting  contours  are  usually  undergrown.  Never¬ 
theless,  RGI  provides  a  good  initial  contour  for  the  following 
evolution  driven  by  active  contour  model. 


III.C.  Background  estimation 

In  the  active  contour  model,  contour  evolution  relies  on 
the  competition  between  the  region  inside  the  contour  (fore¬ 
ground)  and  that  outside  the  contour  (background).  The  pres¬ 
ence  of  structure  noises,  such  as  lymph  nodes,  parenchyma, 
and  localization  markers,  complicates  the  background  in 
mammograms.  RGI  segmentation  provides  not  only  the  ini¬ 
tial  contour,  but  also  a  means  of  estimating  the  effective 
background  surrounding  the  lesion.  In  our  study,  the  effec¬ 
tive  background  is  defined  as  the  set  of  pixels  within  a  given 
distance  d  (pixels)  from  the  circumscribed  rectangle  of  the 
initial  contour,  as  shown  in  Fig.  3. 

Distance  d  plays  an  important  role  in  determining  the  ef¬ 
fective  background.  On  one  hand,  a  large  d  yields  a  large 
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(a)  Cc) 


Fig.  4.  The  illustration  of  determining  the  distance  d:  (a)  a  mammogram 
with  a  series  of  distances  dj,  in  which  the  thick  dashed  rectangle  represents 
the  computer-selected  distance  d\  (b)  B,:  the  ith  background  candidate  cor¬ 
responding  to  dj,  and  (c)  the  ith  net  background  increase.  Background  is 
defined  as  the  set  of  pixels  within  a  given  distance  dj  (pixel)  from  the 
circumscribed  rectangle  of  the  initial  contour. 


region  and  thus  better  statistics  on  the  background.  On  the 
other  hand,  a  small  d  would  not  be  contaminated  by  nearby 
structures.  In  this  study,  an  automatic  scheme  was  developed 
to  determine  the  best  distance  d  from  a  series  of  candidates. 

For  a  series  of  distances  c/„  i=\ , ,L,  two  series  of  re¬ 
gions  can  be  determined,  as  Fig.  4(a)  shows.  One  series  of 
regions  are  background  candidates  B,  [Fig.  4(b)],  and  the 
other  series  are  net  increases  of  background  [Fig.  4(c)], 
where  B^,=B,+i-B„  /=1,  ...,L-1.  Our  method  is  based  on 
the  following  two  principles:  With  the  expansion  of  back¬ 
ground,  (1)  the  mean  gray  value  of  B,-,  i.e.,  mean(B,),  should 
decrease  as  more  areas  with  lower  gray  level  are  included 
and  (2)  the  standard  deviation  of  B^,-,  i.e.,  std(B^,),  should 
not  change  substantially  for  relatively  smooth  background. 
By  monitoring  mean(B,)  and  std(B^,)  with  increasing  B,-,  two 


potential  distance  candidates  are  obtained.  One  candidate  is 
defined  as  the  distance  at  which  mean(B,)  reaches  a  mini¬ 
mum  value,  and  the  other  candidate  is  defined  as  the  distance 
at  which  std(B^,)  demonstrates  the  maximum  increase,  as 
shown  in  Fig.  5.  At  last,  the  final  distance  is  chosen  as  the 
minimum  of  these  two  candidates.  As  for  the  example  in  Fig. 
4,  the  distance  is  automatically  determined  B=110  (pixels). 

III.D.  Background  trend  correction 

Due  to  the  nonuniformity  of  the  background  distribution, 
some  pixels  in  the  background  have  similar  gray  values  as  in 
the  lesion,  which  hinders  the  segmentation  performance  of 
the  active  contour  model.  Thus,  a  two-dimensional  back¬ 
ground  trend  correction  was  employed  prior  to  segmentation. 
The  background  trend  is  estimated  by  fitting  a  two- 
dimensional  surface  with  a  least-squares  method  to  the 
gradual  change  in  the  background  pixel  values  within  the 
extracted  background  estimation  region.  Here,  we  used  a 
first-order  polynomial  function,  i.e.,  f{x,y)  =  a  +  b-x+c-y,  to 
describe  the  two-dimensional  surface  as  higher  order  poly¬ 
nomial  functions  will  estimate  mass  lesion  instead.  Figure  6 
demonstrates  the  significance  of  the  background  trend  cor¬ 
rection  when  a  nonuniform  background  is  present. 

III.E.  Dynamic  stopping  criterion 

To  stop  the  evolution  of  a  contour,  a  predetermined 
threshold  is  often  used.  Various  metrics  can  be  used  to  check 
convergence  of  evolution,  such  as  the  change  of  level  set 
function  cf)  and  the  change  of  length  of  contour.  The  con¬ 
tour  evolution  can  also  be  terminated  when  the  area  inside 
the  contour  differs  from  the  initial  one  by  a  given  value.  In 
our  initial  study,  we  had  defined  a  stopping  criterion  of  rela¬ 
tive  foreground  change  (RFC),  which  is  the  ratio  between  the 
change  of  foreground  and  the  area  of  foreground.  Comparing 
with  the  stopping  criterion  of  change  of  contour  length  used 
in,^*  RFC  has  two  advantages:  (1)  RFC  is  a  relative  measure 
and  thus  is  more  suitable  for  lesions  with  various  sizes  and 
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(b)  (d) 

Fig.  6.  An  example  of  the  effect  of  background  trend  correction  on  segmentation,  (a)  the  original  ROI;  (b)  segmentation  result  of  (a);  (c)  the  processed  ROI 
after  background  trend  correction;  and  (d)  segmentation  result  of  (c). 


(2)  RFC  is  more  computationally  efficient  as  the  acquisition 
of  contour  in  brings  additional  computation.  No  matter 
what  strategy  is  used,  it  is  necessary  to  set  some  threshold  in 
advance.  However,  due  to  varying  sizes  of  lesions,  as  well  as 
sizes  of  background  obtained  from  automatic  background  es¬ 
timation,  it  is  difficult  to  find  a  fixed  parameter  for  control¬ 
ling  convergence. 

In  our  preliminary  work,  we  developed  a  dynamic 
method  to  terminate  contour  evolution  automatically.  In  that 
work,  as  the  contour  evolves,  the  mean  values  of  both  fore¬ 
ground  and  background  will  decrease  gradually.  As  the  fore¬ 


ground  is  generally  more  homogeneous  than  the  background, 
the  rate  of  foreground  mean  change  is  less  than  that  of  back¬ 
ground  mean  change.  However,  as  the  evolving  contour 
crosses  the  lesion  margin,  the  foreground  mean  will  decrease 
faster  than  the  background  mean.  Thus,  during  dynamic  con¬ 
touring,  the  difference  between  the  rate  of  foreground  mean 
change  and  that  of  background  mean  change  is  tracked,  and 
contour  evolution  is  terminated  when  the  decrease  of  fore¬ 
ground  mean  value  is  more  rapid  than  that  of  the  background 
mean  value.  This  method  provides  a  way  to  terminate  con¬ 
tour  evolution  free  of  predefined  threshold.  However,  it  ne- 
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Fig.  7.  The  illustration  of  determining  the  stopping  point.  g{x)  is  a  decreas¬ 
ing  function  defined  on  [0,L]  and  i  e  [0,L]  is  a  moving  point  with  speed  of 
V. 


gleets  the  influence  of  the  sizes  of  both  the  foreground  and 
background,  and  thus  ceases  contour  evolution  earlier  than 
expected. 

In  order  to  address  this  problem,  we  modified  the  previous 
method,  which  we  present  here  in  one  dimension.  As  Fig.  7 
shows,  gW  is  a  decreasing  function  defined  on  the  interval 
[0,L],  and  point  s  is  moving  within  [0,L]  at  the  speed  of  v. 
s  also  splits  [0,L]  into  two  regions.  For  simplicity,  the  region 
[0,s]  is  named  region  1,  and  [s,L]  is  region  2.  Then,  the 
mean  values  of  regions  1  and  2  are; 

I  g{x)dx  I  g{x)dx 

Jo  J  s 


ci  = - 

s 


C2  = 


L  -  s 


The  slope  of  Cj  is 

dci  dci  ds 
dt  ds  dt 


(  n  \ 

g{x)dx 

d 

Jo 

ds 

\  *  / 

^(■s)  -  Cl  - 

-  •  V. 

s 


In  order  to  eliminate  the  influence  of  size,  a  weighted 
difference  between  slope  of  Ci  and  that  of  C2  is  introduced  as 


Au„=  ■ 


dci 

dt 


dC2 

dt 


- -  •  [2  ■  g(s)  -  c(ci  H-  Cj)]  •  V. 

L- s 


(15) 


It  can  be  shown  that  goes  to  zero  at  the  desired  contour 
Sq,  where  g(io)  =  1/2(ciH-C2). 

The  one-dimensional  case,  described  above,  can  be  ex¬ 
tended  to  a  two-dimensional  one.  During  the  contour  evolu¬ 
tion,  the  weighted  difference  between  the  mean  slope  of  fore¬ 
ground  and  that  of  background  is  monitored,  and  the  contour 
evolution  is  terminated  when  the  weighted  slope  difference 
converges  to  zero. 


III.F.  Implementation 

In  order  to  calibrate  parameters  in  the  proposed  segmen¬ 
tation  method,  ten  digitized  screen-film  mammograms 
(SFM)  with  spatial  resolution  of  100 /um  X  100  yum  were 
analyzed.  The  calibrated  segmentation  method  was  then  ap¬ 
plied  to  the  entire  FFDM  database  for  independent  perfor¬ 
mance  evaluation. 

In  our  study,  we  kept  both  \j  and  X2  in  Efi-  0)  to  one  (i.e., 
\i  =  \2=l)  since  the  contribution  of  the  homogeneities  of 
inside  and  outside  the  contour  should  be  equally  considered. 
Other  parameters  in  Eq.  (7)  were  chosen  as  follows:  e=  1  and 
Af=0.1,  where  e  influences  the  Heavyside  function  and  Af 
controls  how  quickly  the  level  set  function  changes.  Note 
that  jji  controls  the  smoothness  of  the  final  contour.  However, 
if  one  wants  to  depict  the  fine  details  of  the  object,  one 
should  choose  a  small  fx.  On  the  contrary,  if  one  wants  to 
obtain  a  smoother  contour,  one  should  set  a  large  jx.  As  some 
of  our  computer-extracted  features,  such  as  spiculation,  char¬ 
acterize  the  fine  details  of  the  lesion  margin,  we  chose  a 
fairly  small  value  of  /x,  i.e.,  0.001  X  1023^,  which  also  allows 
for  the  use  of  the  10-bit  data.  To  ensure  numerical  stability, 
the  coefficient  v  must  satisfy  v-\t<  1/4,^*  so  we  set  v=2  in 
our  study.  The  maximum  number  of  iterations  is  set  to  500. 


Here,  we  use  the  fact  that  v  =  ds/dt-v,  where  v  is  the  outward 
unit  vector.  Similarly,  the  slope  of  C2  is 

dC2  g(.s)  -  C2  ^ 
dt  L  —  s 


■  V. 


Thus,  the  difference  between  these  two  slopes  is 


^  dci  dc2 
dt  dt 


C2 


L-  s 


•  V. 


(14) 


As  the  discussed  above,  as  s  moves  within  the  object,  we 
have  Au  >  0.  As  s  moves  across  the  edge,  Au  will  become 
negative.  When  Au  =  0,  we  have  g{s)  =  sl L-C2  +  {L-sl L)ci 
>  1/2(ci-i-C2),  as  in  general  L-s>s  and  Ci>C2.  However,  if 
only  the  speed  terms  driven  by  image  property  in  Eq.  (7)  are 
considered,  the  evolution  should  stop  at  Sq  such  that  giso) 
=  1 /2(ci-i-C2).  Because  of  the  influence  of  sizes,  s  will  stop 
moving  quickly  if  the  criterion  in  Eq.  (14)  is  used. 


III.G.  Performance  evaluation 


The  performance  of  the  proposed  segmentation  algorithm 
was  assessed  by  comparing  the  computer-delineated  contours 
with  the  outlines  drawn  by  an  expert  breast  radiologist.  Be¬ 
sides  visually  evaluating  the  agreement  of  computer- 
segmented  results  with  radiologist’s  manually  contoured  le¬ 
sion  margins,  a  quantitative  measure  was  used  to  evaluate  the 
segmentation  performance.  Eor  a  particular  lesion,  the  area 
overlap  ratio  (AOR)  between  manual  segmentation  and  com¬ 
puter  segmentation  is  defined  as 


Area(M  D  C) 
Area(M  U  C)  ’ 


(16) 


where  M  is  the  manually  segmented  contour  and  C  is  the 
computer-segmented  contour.  AOR  ranges  from  zero  to  one, 
with  zero  in  the  case  of  no  overlap  and  1.0  in  the  case  of  a 
perfect  match.  Eor  the  entire  database,  a  series  of  AOR 
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Fig.  8.  An  example  of  the  effect  of  level  set  smoothness  to  the  final  segmentation  results.  Left:  Segmentation  without  level  set  smoothness;  Right: 
segmentation  with  level  set  smoothness. 


thresholds  were  obtained  and  at  each  AOR  threshold  one  the 
percentage  of  lesions  “correctly”  segmented  was  calculated 
by  counting  the  number  of  lesions  with  AOR  greater  than 
that  threshold. 

IV.  RESULTS 

IV.A.  Evaluation  of  level  set  smoothness 

In  our  study,  a  new  term  E g=  \ (jjj^dxdy  is  added 
to  the  original  active  contour  model  in,^^  thus  we  initially 
evaluate  the  usefulness  of  this  term.  Two  sets  of  final  con¬ 
tours  were  extracted  from  the  entire  FFDM  database;  one 
was  obtained  with  and  the  other  without.  The  results  show 
that  Eg  cannot  only  provide  a  smoother  contour,  but  also  can 
push  the  contour  closer  to  the  lesion  margin  with  less  itera¬ 
tions,  yielding  a  mean  number  of  iterations  of  160  compared 
to  the  mean  number  of  iterations  of  327  without  E^.  In  the 
example  shown  in  Fig.  8,  the  left  hgure  shows  the  segmen¬ 
tation  result  without  smoothing  level  set  function,  which 
took  500  iterations.  Meanwhile,  for  the  result  with  smooth¬ 
ing  level  set  function  shown  in  the  right  figure,  it  took  only 
248  iterations  to  converge. 

IV.B.  Evaluation  of  dynamic  stopping  criterion 

We  investigated  our  new  stopping  criterion  based  on  the 
weighted  slope  difference  between  foreground  mean  and 
background  mean  (AuJ,  and  compared  it  to  the  unweighted 
slope  difference  method  as  well  as  the  relative  foreground 
change  (RFC).  The  RFC  thresholds  to  terminate  contour  evo¬ 
lution  were  set  as  0.05,  and  0.01,  respectively.  During  the 
evolution,  we  recorded  the  contours  using  these  four  stop¬ 
ping  criteria  and  obtained  the  AOR  with  radiologist’s  out¬ 
lines. 


Figure  9  shows  plots  of  the  fraction  of  correctly  seg¬ 
mented  lesions  at  various  AOR  threshold  for  the  four  stop¬ 
ping  criteria  (Au„,  Au,  RFCo.os,  and  RFCo.oi)  on  the  FFDM 
databases.  For  benign  images,  all  the  criteria  yielded  similar 
segmentation  performances  since  the  initial  contours,  ob¬ 
tained  by  RGI  segmentation,  are  close  to  the  true  lesion  mar¬ 
gins.  However,  as  RGI  segmentation  is  inferior  for  malignant 
lesions,  Au„  does  perform  better  among  all  the  stopping  cri¬ 
teria. 

Table  I  summarizes  the  statistical  comparison  (Holm  t 
test)^"^  among  these  four  criteria,  given  the  mean  and  stan¬ 
dard  deviation  of  AOR  for  each  criterion.  In  terms  of  AOR, 
the  weighted  slope  difference  method  is  statistically  better 
than  the  unweighted  slope  difference  method,  and  the  con¬ 
vergence  rate  at  RFC =0.05  (overall  significant  level 
=  0.05).  However,  we  failed  to  show  a  statistically  significant 
difference  between  the  weighted  slope  difference  method  and 
the  convergence  rate  at  RFC  =  0.01.  Nevertheless,  if  the  num¬ 
ber  of  iterations  is  taken  into  account,  the  mean  number  of 
iterations  for  weighted  slope  difference  is  156,  while  it  is  280 
for  RFCo.oi-  The  weighted  slope  difference  is  more  efficient 
than  RFCo.oi- 

IV.C.  Comparative  evaluation  of  the  segmentation 
method 

The  segmentation  algorithm  was  compared  with  our  pre¬ 
viously  reported  region-growing®  and  RGTbased 

~j 

segmentation  methods.  Figure  10  shows  several  examples 
of  lesion  segmentations  using  these  three  segmentation  meth¬ 
ods.  The  result  of  the  proposed  method  visually  demonstrates 
a  better  agreement  with  the  radiologist’s  outline  of  the  lesion. 

Figure  11  shows  the  fraction  of  lesions  correctly  seg¬ 
mented  at  various  overlap  threshold  levels.  At  the  overlap 
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Fig.  9.  Segmentation  performance  of  four  different  stopping  criteria  in  terms  of  AOR  on  a  clinical  FFDM  database.  In  both  plots,  Ai;„,  is  the  weighted  slope 
difference  between  foreground  mean  value  and  background  mean  value,  in  which  foreground  is  the  area  within  the  evolving  contour  and  background  is  the 
area  outside  contour;  Au  is  the  unweighted  slope  difference  between  these  two  mean  values.  RFCo.oi  stands  for  a  stopping  criterion  that  terminates  contour 
from  evolution  when  the  RFC  is  not  greater  than  0.01.  Similarly,  RFCq.os  stops  the  contour  evolution  when  RFC  is  not  greater  than  0.05.  Left:  evaluated  on 
327  benign  images;  Right:  Evaluated  on  412  malignant  images.  The  results  show  that  the  weighted  slope  difference  is  statistically  superior  to  unweighted  slope 
difference  and  convergence  rate  at  RFC =0.05  on  malignant  images. 


threshold  of  0.4,  for  benign  lesions,  87%  of  the  images  are 
correctly  segmented  with  the  proposed  method,  while  72% 
and  81%  of  the  images  are  correctly  segmented  by  the 
region-growing  and  RGI-based  methods,  respectively.  For 
malignant  lesions,  84%  of  the  images  are  correctly  seg¬ 
mented  with  the  proposed  method,  while  66%  and  67%  of 
the  images  are  correctly  segmented  by  region-growing  and 
RGI-based  methods,  respectively. 

Table  II  gives  the  statistical  comparison  (Holm  t  test)^^  for 
AOR  means  from  the  three  segmentation  methods.  The  im¬ 
provement  of  AOR  with  the  proposed  method  was  found  to 
be  statistically  signihcant  (overall  signihcant  level  q'^=0.05). 


V.  DISCUSSION 

We  developed  a  dual-stage  segmentation  method  to  effi¬ 
ciently  segment  mass  lesions  from  the  parenchymal  surround 
in  FFDM  images.  Our  proposed  method  includes  a  geometric 
active  contour  model,  which  includes  analysis  of  homogene¬ 
ities  both  inside  and  outside  of  the  evolving  contour.  The 
application  of  RGI-based  segmentation  to  provide  initial 
contour  not  only  improves  the  computational  efficiency,  but 
also  provides  a  method  with  which  to  estimate  the  effective 
background  about  the  lesion  and  to  suppress  unrelated  pixel 


Table  I.  Statistical  comparison  of  the  performance  of  four  stopping  criteria  in  the  dual-stage  segmentation  in 
terms  of  AOR,  and  p-values  ai‘e  given  for  the  comparison  of  the  weighted  slope  difference  with  any  other 
stopping  criterion.  The  significant  level  a,  for  the  individual  paired  t  test  is  calculated  using  Holm’s  procedure 
(overall  a;^=0.05).  Same  convention  as  Fig.  9. 


A[)„. 

Av 

RFQi.oi 

^FCo.05 

Benign 

mean+std 

0.61+0.19 

0.61  +  0.19 

0.61+0.19 

0.61+0.19 

p-value 

— 

0.856 

0.801 

0.601 

sig.  lev.  (or,) 

— 

— 

— 

— 

Malignant 

mean+std 

0.59+0.19 

0.53  +  0.20 

0.57+0.19 

0.52+0.20 

p-value 

— 

<0.001 

0.192 

<0.001 

sig.  lev.  (or,) 

— 

0.05 

— 

0.025 

All 

mean+std 

0.60+0.19 

0.57+0.20 

0.59+0.19 

0.56+0.20 

p-value 

— 

0.002 

0.25 

<0.001 

sig.  lev.  (cr,) 

— 

0.05 

— 

0.025 
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(a)  (b)  (c)  (d) 


Fig.  10.  Segmentation  results  for  five  malignant  lesion  examples:  (a)  radiologist’s  outline,  (b)  region-growing,  (c)  RGI-based  segmentation,  and  (d)  the 
proposed  dual-stage  segmentation  method. 
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Fig.  11.  Performance  of  three  different  segmentation  methods  in  terms  of  AOR  on  a  clinical  FFDM  database.  Left:  Evaluated  on  327  benign  images;  Right: 
Evaluated  on  412  malignant  images.  The  results  show  that  the  dual-stage  segmentation  method  is  statistically  superior  to  both  region-growing  and  RGI-based 
method. 


values.  Also,  our  automatic  stopping  criterion  is  lesion- 
specific  and  does  not  rely  on  fixed  iterations. 

As  the  results  show,  the  term  £5  in  the  active  contour 
model  plays  an  important  role  in  effective  and  efficient  seg¬ 
mentation.  As  ||V^||>1,  div[(l-(l/||V</>||)) V</>]  will  evolve 
the  level  set  function  cj)  toward  reducing  ||V(^||,  thus  to 
smooth  <f).  The  larger  the  gradient  magnitude  of  level  set 
function,  the  more  it  will  be  smoothed.  While  as  ||V</i||<l, 
div{[l-(l/||V^||)]V^}  will  evolve  the  level  set  function  to¬ 
ward  increasing  ||V</>||  to  maintain  the  gradient  of  the  level  set 
function  to  some  level.  This  mechanism  ensures  the  level  set 
function,  and  thus  the  final  contour,  is  relatively  smooth. 
Meanwhile,  as  ||V</>||  is  restricted  in  magnitude,  the  fore¬ 
ground  has  the  potential  to  grow  faster. 

It  should  be  noticed  that  the  weighted  slope  difference 
is  always  non-negative  as  long  as  g(x)  is  a  decreasing 

Table  II.  Statistical  comparison  of  the  three  lesion  segmentation  algorithms. 
Performance  is  given  by  average  AOR,  and  p-values  are  given  for  the  com¬ 
parison  of  the  dual-stage  segmentation  with  the  previous  region-growing 
and  RGI-based  method.  The  significant  level  cr,  for  the  individual  paired  t 
test  is  calculated  using  Holm’s  procedure  (overall  Qf^=0.05). 


Dual-Stage 

Region- 

segmentation 

RGI 

growing 

Benign 

mean+std 

0.61+0.19 

0.58+0.19 

0.51  +  0.20 

p-value 

— 

0.01 

<0.001 

seg.  lev.  {aj) 

— 

0.05 

0.025 

Malignant 

mean+std 

0.59+0.19 

0.48+0.20 

0.49+0.20 

p-value 

— 

<0.001 

<0.001 

seg.  lev. 

— 

0.025 

0.05 

All 

mean+std 

0.60+0.19 

0.52+0.20 

0.50+0.20 

p-value 

— 

<0.001 

<0.001 

seg.  lev.  {aj) 

— 

0.05 

0.025 

function.  In  the  active  contour  model,  if  only  the  speed  term 
driven  by  image  property  is  considered,  the  speed  of  contour 
can  be  simplified  as; 

V  =  [(g(i)  -  C2f  -  igis)  -  Ci)^]  •  V, 

=  (ci  -  C2)  •  [2  •  g(i)  -  (ci  H-  C2)]  •  V, 

where  v  is  the  outward  unit  vector.  Inserting  v  into  Eq.  (15), 
we  have; 

Au„  =  - - •  (ci  -  C2)  •  [2  •  g(i)  -  (ci  H-  C2)]^  •  U  0. 

L  -  s 

If  V  is  driven  by  another  image  property,  such  as  edge  infor¬ 
mation,  this  relationship  still  holds.  When  g(s)>l/2(ci 
H-C2),  i.e.,  s  is  within  the  object,  the  contour  will  move  out¬ 
ward  to  the  edge,  thus,  we  have  Av^^,^0.  While  if  g(y) 
<  1/2(ciH-C2),  i.e.,  s  is  out  of  object,  it  will  move  inward  to 
the  edge,  we  will  also  have  Au„S=0.  So  the  weighted  slope 
difference  also  provides  a  general  mechanism  for  terminating 
contour  evolution  with  other  active  contour  models. 

In  this  study,  we  empirically  compared  the  segmentation 
performance  of  the  proposed  method  with  our  previously 
reported  region  growing®  and  RGI-based^  segmentation 
methods.  However,  it  is  impossible  for  us  to  perform  empiri¬ 
cal  comparisons  between  our  method  and  those  reviewed  in 
the  introduction  section,  as  we  do  not  have  codes  of  those 
methods.  Timp’s  method*®  uses  polar  coordinate  and  restricts 
the  mass  sizes  within  a  certain  range;  thus  one  would  expect 
their  method  to  work  better  for  lesions  with  circularlike  mar¬ 
gins.  However,  for  lesions  with  irregular  shapes  or  very  large 
sizes,  their  method  may  have  difficulty.  Our  dual  stage  seg¬ 
mentation  method  is  able  to  handle  this  situation  by  further 
evolving  the  contour  via  the  active  contour  model.  For  the 
fuzzy-set-based  methods  developed  by  Guliato  et  a/.,**  both 
of  them  need  to  present  some  thresholds  such  as  the  gray- 
level  threshold  in  the  first  method  and  the  maximum  allowed 
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difference  between  the  value  of  the  pixel  being  analyzed  and 
the  mean  of  the  subregion  in  the  second  method,  which  pre¬ 
vents  these  methods  from  being  applied  in  a  large  database. 
Their  two  thresholds  were  manually  selected  case  by  case  in 
their  evaluation  using  a  database  with  47  mammograms.  On 
the  other  hand,  our  method  is  flexible  in  that  no  threshold 
needs  to  be  set  in  advance. 

In  our  preliminary  study,  we  compared  two  radiologists’ 
outlines  with  a  digitized  screen-film  mammograms  (SFM) 
database,  which  consisted  of  29  benign  (51  mammograms) 
and  55  malignant  (96  mammograms)  lesions.  At  an  overlap 
threshold  of  0.4,  96.6%  of  lesion  images  were  correctly  seg¬ 
mented  by  one  radiologist  in  comparison  with  the  other.  This 
result  indicates  that  the  radiologists  highly  agreed  on  the 
lesion  margins  for  SFM.  We  could  expect  that  the  radiolo¬ 
gists  would  also  agree  on  the  lesion  margins  for  FFDM  as 
the  manufacturer  has  preprocessed  the  FFDM  images  to 
make  them  appear  to  the  radiologist  as  traditional-looking 
SFM  mammographs. 

When  we  developed  the  proposed  segmentation  algo¬ 
rithm,  the  FFDM  database  was  being  constructed,  so  our 
method  was  initially  calibrated  and  tested  with  the  SFM 
database.  After  building  the  FFDM  database,  we  randomly 
picked  three  groups  of  FFDM  images,  each  of  which  con¬ 
sisted  of  five  benign  and  five  malignant  images,  and  evalu¬ 
ated  the  segmentation  performance  using  the  proposed 
method  calibrated  with  SFM  images.  The  results  were  simi¬ 
lar  with  what  we  had  obtained  with  SFM  images.  Thus,  we 
believe  that  the  parameters  obtained  by  SFM  also  work  with 
FFDM  images,  which  was  subsequently  validated  by  the  in¬ 
dependent  evaluation  with  the  entire  FFDM  database. 

Our  results  could  be  partially  explained  by  the  preprocess¬ 
ing  of  FFDM  images,  which  is  performed  by  the  manufac¬ 
turers.  After  preprocessing,  the  gray-level  range  and  contrast 
of  FFDM  images  become  similar  to  those  of  SFM  images, 
which  ensures  the  possibility  of  applying  parameters  from 
SFM  images  to  FFDM  images  as  gray-level  range  and  con¬ 
trast  are  two  key  components  used  in  our  proposed  lesion 
segmentation  method.  Our  results  also  show  the  robustness 
of  the  proposed  method  as  it  mainly  uses  the  global  informa¬ 
tion  of  images. 


VI.  CONCLUSION 
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APPENDIX 

In  this  part,  we  provide  the  details  of  the  derivation  from 
energy  function  (4)  to  the  associated  Euler-Lagrange  equa¬ 
tion  (5).  For  convenience,  we  restate  Eq.  (4)  here  as 


Ee{ci,C2,4>)  ■■ 


I 


IL  -  4(^(x,y))||V</.(x,y)|| 


+  2  ■  “  l|V0(.«,y)||)^  +  Xi  •  \foix,y) 

-  cip7/^(^(x,y))  H-  X.2  •  l/oUy) 


dxdy. 


(Al) 


We  define  F{(f),V(f),x,y)  as 


F(^,  V^,x,y)  =  (U,4(</.)||V(/>||  H- 1(1  -  ||V(/>||)^  H- Xil/o 
-  Cip//,(<^)  +  \2|/o  -  C2p(l  - //,(0)) . 

(A2) 

Eor  simplicity,  we  have  omitted  the  independent  variables 
(x,y)  of  4>  and  /g.  According  to  calculus  of  variations,  the 
scalar  function  (f)ix,y)  that  minimizes  Ej^Ci,C2, 4>)  solves  the 
PDE; 


In  this  article,  we  present  a  new  lesion  segmentation 
method  based  on  a  geometric  active  contour  model,  which 
includes  an  initial  RGI  segmentation,  background  estimation, 
background  trend  correction,  and  a  dynamic  stopping  crite¬ 
rion.  Evaluation  with  a  large  number  of  EEDM  images  has 
shown  that  the  proposed  method  is  statistically  superior  to 
our  previous  region-growing  and  RGI-based  algorithms  in 
terms  of  overlap  ratios  obtained  in  comparison  with  experts’ 
manual  outlines.  At  an  overlap  threshold  of  0.4,  85%  of  the 
images  are  correctly  segmented  by  the  proposed  method, 
while  only  69%  and  73%  of  the  images  are  correctly  seg¬ 
mented  by  our  previous  region-growing  and  RGI-based 
methods,  respectively. 


d  i  dF\  d  i  dF\  dF  ,  ^ 

—  -  -H—  -  -  —  =  0.  (A3) 

dx  \  d4>j.  /  dy\  d(f)y  /  d4> 

Taking  the  partial  derivative  of  F  with  respect  to  <f>^,  (p^., 
and  (f),  respectively,  we  have: 
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where  S'^=d8ld(j)  and  we  use  the  relation  ||V^||= 
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Similarly,  we  have 
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Inserting  Eqs.  (A4)-(A6)  back  to  (A3),  we  obtain 
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we  finally  obtain  the  compact  form  of  Eq.  (A7)  as 
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Comparison  of  image  segmentation  methods  on  classification  performance  of  FFDM  CAD 


Section  I:  Segmentation  methods 

The  classification  performance  of  two  segmentation  methods  were  investigated  and 
compared  in  this  study. 

1)  Region  growing  methoS^^ 

In  the  region  growing  method,  a  two-dimensional  background  correction  and  histogram 
equalization  are  first  applied  to  the  original  image  data.  Gray  level  thresholding  is 
subsequently  performed  on  the  processed  image  to  yield  contours.  By  monitoring  the  size 
and  shape  of  the  evolving  contour  with  each  incremental  threshold  step,  the  final  lesion 
contour  (i.e.,  lesion  margin),  corresponding  to  an  abrupt  transition  from  small  size  to 
larger  size,  and  from  high  circularity  to  low  circularity,  is  automatically  selected. 

2)  Dual-stage  segmentation^^^ 

In  the  dual-stage  segmentation  method,  a  radial  gradient  index  (RGI)  based  segmentation 
is  used  to  yield  an  initial  contour  close  to  the  lesion  boundary  location  in  a 
computationally  efficient  manner.  This  initial  segmentation  also  provides  a  base  to 
identify  the  effective  circumstance  of  the  lesion  via  an  automatic  background  estimation 
method.  Then  a  region-based  active  contour  model  is  utilized  to  evolve  the  contour  to  the 
lesion  boundary.  This  active  contour  model  relies  on  an  intrinsic  property  of  image 
segmentation  that  each  segmented  region  should  be  as  homogeneous  and  possible  for  an 
image  formed  by  two  regions.  Instead  of  empirically  determined  criteria  such  as  fixed 
iteration  times,  a  dynamic  stopping  criterion  is  implemented  to  terminate  the  contour 
evolution  when  it  reaches  the  lesion  boundary. 

Section  II:  Segmentation  results 

The  performance  of  segmentation  was  initially  evaluated  by  comparing  the  computer 
segmentation  with  manual  outlines  delineated  by  an  expert  breast  radiologist.  Figure  1 
shows  a  malignant  example  of  lesion  segmentation,  which  indicates  that  the  dual-stage 


(a)  (b)  (c) 

Fig.  1  A  malignant  example  of  lesion  segmentation  via  different  methods:  (a)  region  growing,  (b) 
dual-stage  method,  and  (e)  radiologist’s  outline 


Comparison  of  image  segmentation  methods  on  classification  performance  of  FFDM  CAD 


method  visually  demonstrates  a  much  closer  agreement  with  the  radiologist’s  outline  of 
the  lesion. 

Section  III:  Classification  performance 

By  stepwise  feature  selection  with  Wilks  lambda  criterion,  three  features,  including 
normalized  radial  gradient  (NRG)  of  ROI,  NRG  of  lesion  and  gradient  texture,  were 
selected  from  15  features  being  extracted  from  the  lesions  segmented  by  the  dual-stage 
method,  which  yielded  an  AUC  of  0.78.  Margin  sharpness,  gradient  texture,  two  lesion 
margin  spiculation  features  and  the  average  gray  level  were  selected  for  region  growing 
method,  yielding  an  AUC  of  0.72.  The  difference  is  statistically  significant.  Figure  2 
shows  the  ROC  curves  resulting  from  evaluation  of  these  two  groups  of  features. 


Fig.2  ROC  curves  of  leave-one-out  by  lesion  for  the  feature  subset  from  the  dual-stage  segmentation 

method  (solid  line)  and  the  feature  subset  from  the  region  growing  method  (dash  line). 
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Identifying  the  corresponding  images  of  a  lesion  in  different  views  is  an  essential  step  In  Improving 
the  diagnostic  ability  of  both  radiologists  and  computer-aided  diagnosis  (CAD)  systems.  Because  of 
the  nonrIgIdIty  of  the  breasts  and  the  2D  projective  property  of  mammograms,  this  task  Is  not 
trivial.  In  this  pilot  study,  we  present  a  computerized  framework  that  differentiates  between  corre¬ 
sponding  Images  of  the  same  lesion  In  different  views  and  noncorresponding  Images,  I.e.,  Images  of 
different  lesions.  A  dual-stage  segmentation  method,  which  employs  an  Initial  radial  gradient  Index 
(RGI)  based  segmentation  and  an  active  contour  model.  Is  applied  to  extract  mass  lesions  from  the 
surrounding  parenchyma.  Then  various  lesion  features  are  automatically  extracted  from  each  of  the 
two  views  of  each  lesion  to  quantify  the  characteristics  of  density,  size,  texture  and  the  neighbor¬ 
hood  of  the  lesion,  as  well  as  Its  distance  to  the  nipple.  A  two-step  scheme  Is  employed  to  estimate 
the  probability  that  the  two  lesion  Images  from  different  mammographic  views  are  of  the  same 
physical  lesion.  In  the  first  step,  a  correspondence  metric  for  each  pairwise  feature  is  estimated  by 
a  Bayesian  artificial  neural  network  (BANN).  Then,  these  pairwise  correspondence  metrics  are 
combined  using  another  BANN  to  yield  an  overall  probability  of  correspondence.  Receiver  oper¬ 
ating  characteristic  (ROC)  analysis  was  used  to  evaluate  the  performance  of  the  Individual  features 
and  the  selected  feature  subset  In  the  task  of  distinguishing  corresponding  pairs  from  noncorre¬ 
sponding  pairs.  Using  a  FFDM  database  with  123  corresponding  Image  pairs  and  82  noncorre¬ 
sponding  pairs,  the  distance  feature  yielded  an  area  under  the  ROC  curve  (AUC)  of  0.81  ±0.02 
with  leave-one-out  (by  physical  lesion)  evaluation,  and  the  feature  metric  subset,  which  Included 
distance,  gradient  texture,  and  ROI-based  correiation,  yielded  an  AUC  of  0.87 ±0.02.  The  Im¬ 
provement  by  using  multiple  feature  metrics  was  statistically  significant  compared  to  single  feature 
performance.  ©  2008  American  Association  of  Physicists  in  Medicine.  [001:10.1118/1.3005641] 

Key  words:  computer-aided  diagnosis,  full-field  digital  mammography,  correlative  feature  analysis, 
lesion  segmentation,  feature  selection 


I.  INTRODUCTION 

Breast  cancer  Is  a  leading  cause  of  mortality  In  American 
women,  with  an  estimated  182  460  new  cancer  cases  and 
40  480  deaths  In  the  United  States  In  2008.^  Nevertheless, 
between  the  years  1990  to  2003,  there  has  been  a  steady 
decrease  In  the  annual  death  rate  due  to  female  breast 
cancer.^  This  decrease  largely  reflects  Improvements  In  early 
detection  and  treatment.  Currently,  x-ray  mammography  Is 
the  most  prevalent  Imaging  procedure  for  the  early  detection 
of  breast  cancer.^ 

During  mammographic  screening,  multiple  projection 
views,  such  as  cranlocaudal  (CC),  medlolateral  oblique 
(MLO),  and  medlolateral  (ML)  views,  are  usually  obtained. 
Researchers  have  analyzed  Images  from  these  different 
views  to  Increase  the  performance  of  computer-aided  detec¬ 
tion.  Paquerault  et  ai.^  developed  a  two-view  matching 
method  that  computes  a  correspondence  score  for  each  pos¬ 
sible  region  pair  In  CC  and  M  LO  views,  and  merged  It  with 
a  single-view  detection  score  to  Improve  lesion  detectability. 
To  reduce  the  number  of  false  positive  detections,  Zheng 
etai.^  Identified  a  matching  strip  of  interest  on  the  ipsi lateral 
view  based  on  the  projected  distance  to  the  nipple  and 
searched  for  a  region  within  the  strip  and  paired  It  with  the 


original  region.  Engeland  et  a/.®  built  a  cascaded  multiple- 
classifier  system,  in  which  the  last  stage  computes  suspi¬ 
ciousness  of  an  Initially  detected  region  conditional  on  the 
existence  and  similarity  of  a  linked  candidate  region  In  the 
other  view. 

It  has  also  been  well  recognized  that  multiple  views  can 
Improve  the  diagnosis  of  breast  cancer  In  the  computerized 
analysis  of  mammograms, since  different  projections  pro¬ 
vide  complementary  Information  about  the  same  physical  le¬ 
sion.  To  merge  Information  from  Images  of  different  views, 
an  essential  step  Is  to  verify  that  these  Images  actually  rep¬ 
resent  the  same  physical  lesion. 

We  present  a  dual-stage  correlative  feature  analysis  (C  FA ) 
method  to  address  the  task  of  classifying  corresponding  Im¬ 
ages  of  lesions  as  seen  In  different  views.  In  this  method, 
mass  lesions  are  initially  segmented  automatically  from  the 
surrounding  parenchyma.  Then  various  features,  including 
distance,  morphological,  and  textural  features,  are  extracted 
from  the  mass  lesion  on  each  of  the  two  views.  For  a  given 
pair  of  images,  one  from  each  view,  each  pair  of  computer- 
extracted  features  is  merged  through  a  Bayesian  artificial 
neural  network  (BANN)  to  obtain  correspondence  metrics. 
The  correspondence  metrics  are  then  merged  with  a  second 
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BANN  to  yield  an  estimate  of  the  probability  that  the  two 
lesions  on  different  mammographic  images  are  of  the  same 
physical  lesion.  This  CFA  method  is  different  from  conven¬ 
tional  image  registration  methods  in  the  following  two  as¬ 
pects:  (1)  The  task  of  image  registration  is  to  align  two  im¬ 
ages  known  to  represent  the  same  object,  while  CFA  is  to 
assess  the  probability  that  the  given  two  images  represent  the 
same  object.  (2)  The  key  point  of  image  registration  is  to 
determine  a  geometrical  transformation  that  minimizes  some 
cost  functions  defined  by  intensities,  contours,  and  mutual 
information, in  which  various  geometrical  landmarks, 
such  as  control  points  and  inherent  image  landmarks  (nipple, 
curves,  regions  and  breast  skin),^^'^°  are  identified  and 
matched.  The  proposed  CFA  technique  is  feature  based, 
which  is  motivated  by  the  studies  on  fusion  of  two-view 
information  for  computer-aided  detection,^'®  as  well  as  our 
prior  research  on  the  task  of  automated  classification  of 
breast  lesions,  i.e.,  in  the  determination  of  benign  and  malig¬ 
nant  breast  lesions  based  on  computer-extracted  features.^^'^^ 

Differing  from  the  studies  on  computer-aided  detection, 
however,  our  purpose  is  to  identify  the  corresponding  lesions 
from  different  views,  and  ultimately  improve  the  perfor¬ 
mance  of  computer-aided  diagnosis.  Therefore,  the  noncor¬ 
responding  pairs  in  our  study  will  be  lesion-lesion  pairs,  as 
compared  to  the  lesion-parenchyma  or  parenchyma- 
parenchyma  noncorresponding  pairs  in  lesion  detection  task. 
In  a  correspondence  study  between  two  mammographic 
views  for  the  lesion  diagnosis  task,  Gupta  et  al}^  investi¬ 
gated  the  correlation  between  corresponding  texture  features 
from  two  different  views,  and  suggested  that  one  could  in¬ 
clude  features  from  an  additional  view  only  if  they  were  less 
correlated  with  features  from  the  existing  view,  i.e.,  provid¬ 
ing  more  complementary  information.  Our  study,  however, 
does  not  discuss  methods  to  merge  information  from  differ¬ 
ent  views,  but  rather  focuses  on  classifying  the  correspon¬ 
dence  between  lesions  instead. 

II.  MATERIALS  AND  METHODS 

The  main  aspect  of  the  proposed  correlative  feature  analy¬ 
sis  includes  automatic  lesion  segmentation,  computerized 
feature  extraction,  feature  selection,  and  an  estimation  of  the 
probability  that  two  given  images  represent  the  same  physi¬ 
cal  lesion.  Figure  1  shows  the  schematic  diagram  of  the  pro¬ 
posed  method. 

II.A.  Database 

The  full-field  digital  mammography  (FFDM)  database  in 
our  study  consists  of  135  biopsyproven  mass  lesions  ac¬ 
quired  at  the  University  of  Chicago  Flospitals,  in  which  le¬ 
sions  were  collected  under  an  approved  institutional  review 
board  (IRB)  protocol.  Of  the  135  lesions,  67  are  benign  with 
123  mammograms  and  68  are  malignant  with  139  mammo¬ 
grams.  All  the  images  were  obtained  from  GE  Senographe 
2000D  systems  (GEM  edical  Systems  M  ilwaukee,  Wl)  with 
a  spatial  resolution  of  100x100  ytm^  in  image  plane.  The 
masses  were  identified  and  outlined  by  an  expert  breast  ra¬ 
diologist  based  on  visual  criterion  and  biopsy-proven  reports. 
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Based  on  the  correspondence  of  lesions  identified  by  the  ra¬ 
diologist,  we  constructed  123  corresponding  pairs  and  82 
noncorresponding  pairs.  Each  pair  consists  of  a  CC  view  and 
a  M  L  view.  Figure  2  shows  an  example  case  with  multiple 
lesions  seen  on  mammograms  in  CC  and  M  L  views.  Consid¬ 
ering  the  most  realistic  scenario  of  lesion  mismatch  in  clini¬ 
cal  practice,  the  noncorresponding  pairs  were  constructed 
from  cases  of  the  same  patients  but  different  physical  lesions. 
Since  in  our  database  only  28  patients  had  two  or  more  le¬ 
sions  in  the  same  breast,  the  noncorresponding  dataset, 
which  includes  all  possible  lesion  combinations  from  the  dif¬ 
ferent  views,  is  limited.  Table  I  lists  the  detailed  information 
regarding  the  corresponding  and  noncorresponding  datasets. 


II. B.  Lesion  segmentation 

In  our  study,  a  dual-stage  method,^®  on  which  we  have 
already  reported,  was  employed  to  automatically  extract  le¬ 
sions  from  the  normal  breast  tissue.  In  this  method,  a  radial 
gradient  index  (RGI)  based  segmentation^^  is  used  to  yield 
an  initial  contour  in  a  computationally  efficient  manner.  This 
initial  segmentation  also  provides  a  base  to  identify  the  ef¬ 
fective  circumstance  of  the  lesion  via  an  automatic  back¬ 
ground  estimation  method.  Then  a  region-based  active  con¬ 
tour  model is  utilized  to  evolve  the  contour  further  to  the 
lesion  margin.  The  active  contour  model  relies  on  an  intrinsic 
property  of  image  segmentation  in  that  each  segmented  re¬ 
gion  (i.e.,  the  lesion  region  and  the  parenchymal  background 
region)  should  be  as  homogeneous  as  possible.  Thus,  the 
contour  evolution  tries  to  minimize  the  following  energy 
function: 

£  (Cl,  C2,  C )  =  /X  ■  L  ength(C )  -F  V  ■  ^  J  (1  - 1| V 0tll)^  dxdy 

+  ^r  |fo(x,y) -cipdxdy 

J  inside(C) 

+  ^2-  f  |fo(x,y)-C2pdxdy,  (1) 

d  out5ide(C) 

where  yt  3=0,  v^O,  Xi,  X2>0  are  fixed  weight  parameters,  C 
is  the  evolving  contour,  and  Length(C)  is  a  regularizing  term 
that  prevents  the  final  contour  from  converging  to  a  small 
area  due  to  noise,  fl  represents  the  entire  image  space  and 
/ii(l-||V^t||)^dxc(y  is  an  additional  regularizing  term  that 
provides  a  smoother  contour  and  pushes  the  contour  closer  to 
the  lesion  margin  with  less  iterations.  Ci  and  C2  are  mean 
values  inside  and  outside  of  C,  respectively.  The  minimiza¬ 
tion  of  this  energy  function  can  be  achieved  by  level  set 
theory^^  and  Calculus  of  Variations,  in  which  the  two- 
dimensional  evolving  contour  C  is  represented  implicitly  as 
the  zero  level  set  of  a  three-dimensional  function  </>(x,y),  i.e., 
C={(x,y)  efl:</>(x,y)=0}.  Instead  of  empirically  deter¬ 
mined  criteria  such  as  fixed  iteration  times,  a  dynamic  stop¬ 
ping  criterion  is  implemented  to  automatically  terminate  the 
contour  evolution  when  it  reaches  the  lesion  boundary. 
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Fig.  1.  Schematic  diagram  of  the  proposed  correiative  feature  analysis. 


II.C.  Computerized  feature  extraction 

In  this  study,  our  primary  interest  is  to  investigate  the 
potential  usefulness  of  various  computer-extracted  features 
in  the  task  of  differentiating  corresponding  image  pairs  from 
noncorresponding  ones.  Features  in  this  study  are  grouped 
into  three  categories:  (I)  margin  and  density  features,  (II) 
texture  features  based  on  gray-level  co-occurrence  matrix 
(GLCM ),  and  (III)  a  distance  feature.  The  features  in  the  first 
two  categories  have  been  described  in  detail  elsewhere^^'^^ 
and  are  only  summarized  here. 

II.C.I.  Margin  and  density  features 

M  argin  and  density  of  a  mass  are  two  important  properties 
used  by  radiologists  when  assessing  the  probability  of  malig¬ 
nancy  of  mass  lesions.  The  margin  of  a  mass  can  be  charac¬ 
terized  by  its  sharpness  and  spiculation.  The  margin  sharp¬ 
ness  is  calculated  as  the  average  of  the  gradient  magnitude 
along  the  margin  of  the  mass.^^  The  margin  spicuiation  is 
measured  by  the  full  width  at  half  maximum  (FWHM )  of  the 
normalized  edge-gradient  distribution  calculated  for  a  neigh¬ 
borhood  of  the  mass  with  respect  to  the  radial  direction,  and 
by  the  normalized  radial  gradient  (NRG).^^  Three  features 
were  extracted  to  characterize  different  aspects  of  the  density 
of  a  lesion.  Gradient  texture  is  the  standard  deviation  of  the 
gradient  within  a  mass  lesion.  Average  gray  vaiue  is  obtained 
by  averaging  the  gray  level  values  of  each  pixel  within  the 
segmented  region  of  mass  lesion,  and  contrast  measures  the 
difference  between  the  average  gray  level  of  the  segmented 
region  and  that  of  the  surrounding  parenchyma.  Furthermore, 
an  equivaient  diameter  feature  was  also  used  in  this  study, 
which  is  defined  as  the  diameter  of  a  circle  yielding  the  same 
area  as  the  segmented  lesion. 


II.C.2.  Texture  features 

The  calculation  of  texture  features  in  our  study  is  based 
on  the  gray-level  co-occurrence  matrix  (GLCM  For 

an  image  with  G  gray  levels,  the  corresponding  GLCM  is  of 
size  G  xG,  where  each  element  of  the  matrix  is  the  joint 
probability  (Pr, «(/,)))  of  the  occurrence  of  gray  levels  /'  and  j 
in  two  paired  pixels  with  an  offset  of  r  (pixels)  along  the 
direction  6>  in  the  image. 

Fourteen  texture  feature  were  extracted  from  the  GLCM 
matrix,  including  contrast,  correiation,  difference  entropy, 
difference  variance,  energy,  entropy,  homogeneity,  maximum 
correiation  coefficient,  sum  average,  sum  entropy,  sum  vari¬ 
ance,  variance,  and  two  information  measures  of  correiation. 
These  features  quantify  different  characteristics  of  a  lesion, 
such  as  homogeneity,  gray-level  dependence,  brightness, 
variation,  and  randomness. 

In  our  study,  texture  features  were  extracted  from  the  le¬ 
sion  and  the  associated  region  of  interest  (ROI).  A  ROI  in¬ 
cludes  a  lesion  and  its  surrounding  neighborhood,  which  was 
determined  by  an  automatic  estimation  method  developed  in 
our  prior  study.^”  FI  ere,  an  effective  neighborhood  is  defined 
as  the  set  of  pixels  within  a  distance  d  (pixels)  from  the 
circumscribed  rectangle  of  the  segmented  lesion,  as  shown  in 
Fig.  3.  It  should  be  noted  that  this  neighborhood  estimation 
is  similar  to  that  used  earlier  in  the  lesion  segmentation, 
however,  here  the  ROI  is  centered  to  the  edge  of  the  seg¬ 
mented  lesion.  Furthermore,  a  two-dimensional  linear  back¬ 
ground  trend  correction  was  employed  after  the  ROI  extrac¬ 
tion  to  eliminate  the  low-frequency  background  variations  in 
the  mammographic  region.^® 

For  each  region,  four  GLCMs  were  constructed  along 
four  different  directions  of  0°,  45°,  90°,  and  135°,  and  a 
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Fig.  2.  An  example  of  two  lesions  in  the  same  breast  as  seen  in  CC  view 
(left)  and  M  L  view  (right).  The  arrow  indicates  the  correspondence  of  the 
same  physical  lesion  in  different  views. 


nondi recti onal  GLCM  was  obtained  by  summing  aii  the  di- 
rectionai  GLCMs.  Texture  features  were  computed  from 
each  nondi  recti  onai  GLCM  ,  resuiting  in  a  totai  of  28  texture 
features.  To  avoid  sparse  GLCM  s  for  smaiier  ROIs,  the  gray 
levei  range  of  aii  the  image  data  was  scaied  down  to  6  bits, 
resuiting  in  GLCM  s  of  size  64x64.  The  offset  r  was  empiri- 
caiiy  determined  to  be  16  (pixeis). 

II.C.3.  Distance  feature 

In  ciinicai  practice,  radioiogists  commoniy  use  the  dis¬ 
tance  from  the  nippieto  the  center  of  a  iesion  to  correiate  the 
lesion  in  different  views.^'^  It  is  generally  believed  that  this 
distance  remains  fairly  constant.  Thus,  a  distance  feature  in 
our  study  measures  the  Euclidean  distance  between  the 
nipple  location  and  the  mass  center  of  the  lesion.  Figure  4 
shows  the  high  correlation  between  the  distance  features  of 
the  same  lesions  in  CC  and  ML  views,  with  a  correlation 
coefficient  of  0.88.  For  this  figure,  the  nipple  locations  were 
manually  identified. 

In  mammographic  images,  nipple  markers  are  commonly 
used.  These  present  as  bright  markers  on  the  mammograms 
(as  shown  in  Fig.  5),  and,  thus,  an  automatic  nipple  localiza¬ 
tion  scheme  was  developed  to  locate  those  markers.  The 
scheme  includes  several  processing  stages.  Initially,  gray- 
level  thresholding  is  employed  to  the  entire  mammogram  to 


Table  I.  The  number  of  lesion/image  pairs  in  corresponding  and  noncorre¬ 
sponding  datasets.  The  noncorresponding  pairs  were  constructed  from  cases 
of  the  same  breasts  but  different  physicai  iesions. 


Corresponding  dataset 

Noncorresponding  dataset 

Benign 

Images 

112 

72 

Lesions 

56 

39 

M  alignant 

Images 

134 

64 

Lesions 

67 

19 

M  ixed 

Images 

- 

28 

Lesions 

— 

14 
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extract  the  breast  region  from  the  air  background.  Then,  an¬ 
other  gray-level  threshold  is  applied  to  the  breast  region, 
yielding  several  nipple  marker  candidates.  The  breast  skin 
boundary  is  obtained  by  subtracting  a  morphologically 
eroded^®  breast  region  from  its  original  region.  To  reduce  the 
number  of  falsely  identified  nipple  markers,  area  and  circu¬ 
larity  constraints  are  imposed  on  each  candidate,  and  those 
candidates  with  area  within  a  given  range  and  circularity 
above  a  certain  threshold  are  kept  for  the  final  step.  The  area 
range  and  circularity  threshold  were  empirically  determined 
with  ten  randomly  selected  images  in  this  study.  The  nipple 
marker  is  finally  chosen  as  the  one  closest  to  the  breast 
boundary.  For  those  cases  in  which  there  is  no  nipple  marker 
or  the  marker  is  neglected  erroneously  by  the  above  scheme, 
the  nipple  location  is  roughly  estimated  as  the  point  on  the 
breast  skin  boundary  with  the  largest  distance  to  the  chest 
wall. 


Fig.  4.  The  correlation  between  distance  features  of  the  same  lesions  in  CC 
and  M  L  views.  The  distance  feature  is  defined  as  the  Euclidean  distance 
between  the  nipple  location  and  the  mass  center  of  the  lesion.  Here,  the 
nipple  location  is  manually  identified. 
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Fig.  5.  Two  examples  of  nipple  markers.  Nipple  markers  are  bright  spots 
close  to  the  breast  skin  boundary,  as  indicated  by  arrows. 


II. D.  Feature  selection  and  classification 

For  each  pairwise  set  of  features  extracted  from  iesions  in 
different  views,  a  Bayesian  Artificiai  Neurai  Network 
(BANN)  ciassifier^^'^”  was  empioyed  to  merge  each  feature 
pair  into  a  correspondence  metric,  which  is  an  estimate  of  the 
probabiiity  that  the  two  iesion  images  are  of  the  same  physi- 
cai  iesion,  i.e.,  stage  /  as  shown  in  Fig.  1.  For  exampie,  Fig. 
6(a)  shows  the  distributions  of  three  features  (distance,  di¬ 
ameter,  and  texture)  generated  from  breast  iesions  taken  in 
different  views  for  corresponding  and  noncorresponding 
datasets.  The  histograms  in  Fig.  6(b)  demonstrate,  for  the 
corresponding  and  noncorresponding  datasets,  the  distribu¬ 
tion  of  these  correspondence  metrics  output  from  the  first 
BANN. 

Linear  stepwise  feature  seiection^^  with  Wiiks  iambda  cri¬ 
terion  was  empioyed  on  aii  feature-based  correspondence 
metrics  to  seiect  a  subset  of  metrics  for  the  finai  task  of 
distinguishing  corresponding  pairs  from  noncorresponding 
ones.  Note  that  instead  of  using  iesion  features  directiy,  the 
correspondence  metrics  obtained  from  the  first  BANNs  are 
used  as  inputs  in  the  feature  seiection.  BANNs  were  then 
retrained  with  the  seiected  correspondence  metrics  to  yieid 
an  overaii  estimate  of  probabiiity  of  correspondence,  i.e.,  the 
second  BANN  stage  as  shown  in  Fig.  1. 


II. E.  Evaluation 

Receiver  operating  characteristic  (ROC)  anaiysis^^'^^  was 
used  to  assess  the  performance  of  the  individuai  feature- 
based  correspondence  metrics  and  the  overaii  performance  in 
the  task  of  distinguishing  corresponding  image  pairs  from 
noncorresponding  ones.  The  area  under  the  maximum 
iikeii hood-estimated  binormai  ROC  curve  (AUC)  was  used 
as  an  index  of  performance.  ROCKIT  software  (version  1.1 
b,  avaiiabie  at  http://xray.bsd. uchicago.  edu/kri/KRL_ROC/ 
software_index6.htm)^^  was  used  to  determine  the  p  vaiue  of 
the  difference  between  two  AUC  vaiues,  and  the  Hoim  t 
test^^  for  muitipie  tests  of  significance  was  empioyed  to 
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evaiuate  the  statisticai  significance.  Leave-one-out  by  iesion 
anaiysis  was  used  in  aii  performance  evaiuations.  This 
method  removes  aii  images  of  a  iesion  whiie  training  with  aii 
other  images.  The  trained  ciassifier  is  then  run  on  the  images 
of  the  iesion  removed.  In  the  case  of  correspondence  analy¬ 
sis,  images  of  all  pairs,  both  corresponding  and  noncorre¬ 
sponding  pairs,  are  removed  in  the  training  to  eliminate  bias. 

III.  RESULTS 
III.A.  Segmentation 

Figure  7  shows  two  examples  of  lesion  segmentation  us¬ 
ing  the  dual-stage  segmentation  method.  A  measure  of  area 
overlap  ratio  (AOR)  was  used  to  quantitatively  evaluate  the 
segmentation  performance,  which  is  defined  as  the  intersec¬ 
tion  of  human  outline  and  computer  segmentation  over  the 
union  of  them.  At  the  overlap  threshold  of  0.4,  81%  of  the 
images  are  correctly  segmented. 

III.B.  Nipple  localization  method 

Figure  8  shows  the  correlation  between  distance  features 
calculated  with  manually  identified  nipples  and  those  calcu¬ 
lated  with  computer-identified  nipples.  These  two  distance 
features  are  highly  correlated  with  correlation  coefficient  of 
0.996  (p-ua/ue<10“^).  Both  of  these  two  distance  features 
have  an  AUC  value  of  0.81  ±0.02  in  the  task  of  distinguish¬ 
ing  between  corresponding  and  noncorresponding  image 
pairs. 

III.C.  Performance  of  single-feature  correspondence 
metrics 

We  calculated  the  correlation  coefficient  r  for  the  corre¬ 
sponding  dataset,  the  r'  for  the  noncorresponding  datasets, 
and  their  associated  p-vaiue  for  features  extracted  from  two 
view  images.  Table  II  shows  the  results  for  features  with  r 
3=0.5.  It  also  shows  theAUC  values  and  the  associated  stan¬ 
dard  errors  (se)  representing  the  performance  of  the  corre¬ 
spondence  metrics  of  these  individual  features  in  the  task  of 
differentiating  the  corresponding  lesion  pairs  from  noncorre¬ 
sponding  ones,  with  the  lesions  automatically  delineated  by 
the  segmentation  algorithm.  The  results  show  that  all  three 
categories  have  potential  for  the  classification  task.  The  re¬ 
sults  also  show  that  the  performance  of  pairwise  image  clas¬ 
sification  as  learned  by  a  BANN  is  determined  by  both  the 
correlation  of  corresponding  pairs  and  that  of  noncorre¬ 
sponding  pairs. 

We  also  investigated  the  effect  of  lesion  segmentation  on 
the  performance  of  each  individual  feature-based  correspon¬ 
dence  metric.  Table  III  shows  theAUC  values  and  the  asso¬ 
ciated  standard  error  (se)  of  the  18  features  extracted  from 
lesions  delineated  by  a  radiologist  and  by  the  dual-stage  seg¬ 
mentation  algorithm,  respectively.  Also  shown  are  the  95% 
confidence  intervals  (C.  I.)  of  the  difference  of  A UCs  ob¬ 
tained  from  radiologist-outlined  lesions  (AUCr)  and  the 
computer-segmented  lesions  (AUCc),  i.e.,  AAUC=AUCfi 
-AUCc.  For  5  of  the  18  features,  manual  segmentation 
yielded  statistically  significant  higherAUC  values  than  com- 
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Fig.  6.  (a)  The  scatter  plots  of  three  features  (distance,  diameter,  and  texture)  generated  from  lesions  seen  on  CC  and  M  L  views,  (b)  The  distribution  of  the 
output  correspondence  metrics  of  these  features  obtained  from  the  first  BA  N  N  stage. 


puter  segmentation  (overall  significant  level  a'^=0.05),^^  and 
we  failed  to  show  significant  differences  between  manual 
segmentation  and  computer  segmentation  for  the  remaining 
features. 


III.D.  Multiple  features  performance 

Two  sets  of  individual  feature-based  correspondence  met¬ 
rics  were  selected  by  stepwise  feature  selection^^— one  set 
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Fig.  7.  Segmentation  results  for  a  benign  lesion  and  a  malignant  lesion.  The  solid  lines  in  the  upper  four  images  depict  the  lesion  margin  as  outlined  by  a 
radiologist,  and  the  solid  lines  in  the  bottom  four  images  are  segmentation  results  from  our  previously-reported  automatic  dual-stage  method  (Ref.  20).  (a)  CC 
view  of  a  benign  lesion,  (b)  the  corresponding  M  L  view  of  the  benign  lesion,  (c)  CC  view  of  a  malignant  lesion,  and  (d)  the  corresponding  M  L  view  of  the 
malignant  lesion. 


for  each  of  the  two  segmentation  methods,  as  shown  on 
Table  IV.  The  subset  selected  from  the  feature-based  corre¬ 
spondence  metrics  based  on  manually  segmented  lesions  in¬ 
cluded  distance  (Fmj),  equivalent  diameter  (Fi^^),  and  gra¬ 
dient  texture  The  subset  selected  from  computer- 

segmented  lesions  included  distance  ROI-based 

correlation  {F  and  gradient  texture  (F,  i).  The  leave-one- 
out  (by  lesion)  validation  using  BANN  to  merge  the  selected 


Distance  from  manual  nipple  identification  (mm) 


Fig.  8.  The  correlation  between  distance  features  calculated  from  manually 
identified  nipple  locations  and  those  calculated  from  computer-identified 
nipple  locations. 


correspondence  metrics  yielded  an  AUC  of  0.89  for  manual 
segmentation  and  0.87  for  computer  segmentation,  respec¬ 
tively.  We  failed  to  show  a  statistically  significant  difference 
between  the  performances  of  these  two  metric  subsets  (p 
=0.35).  The  improvement  by  using  multiple-feature-based 
correspondence  metrics  was  statistically  significant  com¬ 
pared  to  that  of  single  feature-based  correspondence  metric 
performance,  as  shown  in  Table  IV. 

Since  the  distance  feature  performed  best  among  the  in¬ 
dividual  features  for  differentiating  corresponding  and  non¬ 
corresponding  image  pairs,  we  evaluated  the  performance  of 
the  proposed  correlative  feature  analysis  method  with  the 
distance  feature  excluded.  Using  the  remaining  17  features 
extracted  from  the  computer-segmented  lesions,  a  feature- 
based  correspondence  metric  subset  was  obtained  by  step¬ 
wise  feature  selection,  which  included  equivalent  diameter 
(F  1^3),  ROI-based  correlation  (F and  ROI-based  sum  of 
variance  (F„  14).  The  leave-out-out  (by  lesion)  validation  us¬ 
ing  BANN  yielded  an  AUC  of  0.71  ±0.03.  The  difference  as 
compared  to  the  performance  of  distance  feature  is  statisti¬ 
cally  significant  (p  =0.005).  This  result  indicates  that  the  dis¬ 
tance  feature  is  dominant  but  not  sufficient  for  the  overall 
performance  of  the  proposed  method. 

IV.  DISCUSSION 

In  this  study,  we  presented  a  correlative  feature  analysis 
framework  to  assess  the  probability  that  a  given  pair  of  two 
mammographic  images  is  of  the  same  physical  lesion.  Our 
results  demonstrate  that  this  framework  has  potential  to  dis¬ 
tinguish  between  corresponding  and  noncorresponding  lesion 
pairs.  It  is  very  important  to  note  that  our  method  is  feature 
based,  which  employs  two  BANN  classifiers  to  estimate  the 
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Table  II.  Performance  of  the  correspondence  metrics  from  computer-extracted  lesion  features  that  yielded  r 
saO.5  in  differentiating  corresponding  image  pairs  from  noncorresponding  ones,  r  is  the  correlation  coefficient 
for  the  corresponding  dataset  and  r'  is  for  the  noncorresponding  dataset.  The  value  after  ‘‘±"  is  the  standard 
error  (se)  associated  with  each  AUC. 


Corresponding  pairs 

Noncorresponding  pairs 

r 

p  value 

r' 

p  value 

AUC±se 

1.  Density  and  morph,  features 

F/y.  Gradient  texture 

0.53 

<0.001 

0.27 

0.01 

0.56±0.03 

Fi  2'.  Average  gray  level 

0.58 

<0.001 

-0.10 

0.39 

0.54  ±0.03 

F,  2.  Equivalent  diameter 

0.62 

<0.001 

0.14 

0.22 

0.66±0.03 

II.  Texture  features 

*  Lesion  based 

Fil  l'.  Correlation 

0.56 

<0.001 

0.13 

0.25 

0.65±0.03 

Fiif.  Info,  corn  1 

0.50 

<0.001 

0.06 

0.61 

0.67±0.03 

Fiiy  Info,  corn  2 

0.53 

<0.001 

0.09 

0.40 

0.67±0.03 

Fiiy.  M  ax.  corn 

0.53 

<0.001 

0.11 

0.35 

0.66±0.03 

**  ROI  based 

F,iy.  Contrast 

0.58 

<0.001 

0.16 

0.15 

0.54  ±0.03 

Fi,y.  Correlation 

0.67 

<0.001 

0.24 

0.03 

0.56±0.03 

F Ilf.  Diff.  variance 

0.61 

<0.001 

0.20 

0.07 

0.53±0.03 

Fiif  Entropy 

0.51 

<0.001 

0.15 

0.17 

0.56±0.03 

Fiif  Info,  corn  1 

0.62 

<0.001 

0.16 

0.15 

0.61±0.03 

F„  io:  Info,  corn  2 

0.62 

<0.001 

0.14 

0.21 

0.57±0.03 

Fu  n:  M  ax.  corn 

0.61 

<0.001 

0.11 

0.33 

0.55±0.03 

Fu  n-  Sum.  Average 

0.63 

<0.001 

0.27 

0.01 

0.59±0.03 

Fun'.  Sum.  Entropy 

0.53 

<0.001 

0.16 

0.15 

0.57±0.03 

Fii  u.  Sum.  Variance 

0.61 

<0.001 

0.41 

<0.001 

0.50±0.03 

III.  Distance  feature 

Fiiif  Distance 

0.88 

<0.001 

0.23 

0.04 

0.81±0.02 

Table  III.  Performance  of  18  single  feature-based  correspondence  metrics  obtained  from  radiologist-outlined 
(AUCr)  and  computer-segmented  (AUCc)  lesions,  respectively.  The  value  after  ‘'±"  is  the  standard  error  (se) 
associated  with  each  AUC.  The  two-tailed  p-value  and  95%  C.l.  of  AAUC  were  calculated  by  ROCKIT.  The 
"Sig.  level”  column  represents  the  significance  level  of  individual  tests  adjusted  with  Holm  t  test  (overall 
significant  level  a^=0.05)  and  the  tests  with  asterisks  (*)  indicate  significant  difference  using  the  adjusted 
significance  level.  The  features  have  the  same  convention  as  Table  II. 


Feature 

AUC„±se 

AUCc±se 

p  value 

Sig.  level 

95%  C.l.  Of  AAUC 

Fi.i 

0.65  ±0.03 

0.56  ±0.03 

0.04 

0.0045 

[0.004,  0.20] 

Fi.2 

0.53  ±0.03 

0.54  ±0.03 

0.76 

— 

[-0.07,0.05] 

0.78  ±0.03 

0.66  ±0.03 

0.001 

0.0031 

[0.05,  0.19] 

Fii.i 

0.71  ±0.03 

0.65  ±0.03 

0.06 

— 

[-0.01,0.13] 

Fii.2 

0.68  ±0.03 

0.67  ±0.03 

0.66 

— 

[-0.05,0.09] 

’=11.3 

0.69  ±0.03 

0.67  ±0.03 

0.48 

— 

[-0.04,0.09] 

’=  11,4 

0.70  ±0.03 

0.66  ±0.03 

0.20 

— 

[-0.02,0.11] 

’=11.5 

0.57  ±0.03 

0.54  ±0.03 

0.30 

— 

[-0.03,0.10] 

’=  11,6 

0.61  ±0.03 

0.56  ±0.03 

0.01 

0.0042 

[0.01,  0.10] 

’=11.1 

0.61  ±0.03 

0.53  ±0.03 

0.009 

0.0038 

[0.02,  0.15] 

’=  11,8 

0.58  ±0.03 

0.56  ±0.03 

0.44 

— 

[-0.03,0.07] 

F* 

II  9 

0.69  ±0.03 

0.61  ±0.03 

0.002 

0.0036 

[0.03,  0.14] 

F* 

0.65  ±0.03 

0.57  ±0.03 

4X10-^ 

0.0029 

[0.04,  0.13] 

F*' 

0.66  ±0.03 

0.55  ±0.03 

<10-= 

0.0028 

[0.06,  0.15] 

’=  11,12 

0.62  ±0.03 

0.59  ±0.03 

0.34 

— 

[-0.03,0.09] 

’=  11,13 

0.58  ±0.03 

0.57  ±0.03 

0.90 

— 

[-0.05,0.05] 

F* 

0.59  ±0.03 

0.50  ±0.03 

0.001 

0.0031 

[0.05,  0.18] 

’=  111,1 

0.81  ±0.02 

0.81  ±0.02 

0.73 

— 

[-0.01,0.01] 
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Table  IV.  Performances  of  the  overall  correlative  feature  analysis  method  using  leave-one-out  (by  lesion) 
validation,  as  well  as  the  comparison  with  the  distance  feature  alone.  This  tabie  also  shows  the  comparison 
between  the  overaii  performances  of  merged  features  obtained  from  radiologist-outlined  and  computer- 
segmented  lesions.  Same  convention  asTable  III. 


Lesion  segmentation 

Feature  set 

AUC±se 

p  vaiue 

95%  C.l.  of  AAUC 

Radioiogist  outiined 

^  111,1 

^  111,1:  F  1,3'  ^  1,1 

0.81  ±0.02 

0.89  ±0.02 

} 

3  X  lO""* 

[0.04,  0.12] 

} 

0.35 

[-0.02,0.06] 

Computer  segmented 

^  111,1'  F :i,6'  ^  1,1 
^  111,1 

0.87  ±0.02 

0.81  ±0.02 

} 

0.01 

[0.01,  0.08] 

relationships  (linear  or  nonlinear)  between  computer- 
extracted  features  of  a  lesion  in  different  views.  The 
supervised-learning  manner  not  only  makes  the  relationship 
flexible  to  each  feature,  but  also  avoids  the  sophisticated 
geometrically  deformable  models  and  the  corresponding 
computationally  demanding  optimizations  that  are  used  in 
geometric  breast  registrations.™'^^ 

In  our  study,  we  excluded  those  features  characterizing 
subtle  information  of  a  lesion,  such  as  spiculation,  margin 
sharpness,  and  normalized  radial  gradient  (NRG).  These  fea¬ 
tures  have  been  used  in  the  task  of  distinguishing  between 
benign  and  malignant  lesions  for  mammographic  images.®'^^ 
However,  as  the  lesion  details  are  usually  sensitive  to  posi¬ 
tions,  it  is  expected  that  the  associated  features  are  less  cor¬ 
related  in  different  views.  Nevertheless,  our  ultimate  aim  is 
to  improve  the  diagnostic  performance  of  CAD  systems  with 
multiple  images,  in  which  complementary  information  pro¬ 
vided  by  different  images  are  desired:  therefore,  those  fea¬ 
tures  corresponding  to  lesion  details  would  be  used  in  the 
later  step  of  the  overall  CAD  scheme  for  differentiating  be¬ 
tween  malignant  and  benign  lesions. 

in  addition,  as  shown  in  Table  III,  improving  lesion  seg¬ 
mentation  can  improve  the  performance  of  the  computer  in 
differentiating  corresponding  and  noncorresponding  image 
pairs.  This  is  expected  since  more  accurate  segmentation 
yields  more  reliable  computer-extracted  features  with  which 
to  characterize  the  lesion  and  the  two-view  correspondence. 

A  two-stage  procedure  was  employed  to  address  the  prob¬ 
lem  of  estimating  the  probability  of  correspondence  for  a 
pair  of  lesion  images  in  different  views.  Stage  I  deals  with 
the  pairwise  features  and  estimates  the  probability  of  corre¬ 
spondence  based  on  individual  lesion  features.  Stage  II 
merges  the  correspondence  metrics  estimated  in  stage  I  from 
various  individual  lesion  features  to  yield  an  overall  prob¬ 
ability  of  correspondence.  To  illustrate  the  superiority  of  the 
proposed  two-stage  method  to  a  one-stage  method  that  com¬ 
bines  the  multiple  paired  features  directly,  we  compared  the 
performances  of  the  two  methods  with  the  four  features  of 
distance,  lesion  equivalent  diameter,  lesion-based  correla¬ 
tion,  and  lesion-based  information  correlation,  all  of  which 
performed  best  among  the  18  individual  features  extracted 
from  computer-segmented  lesions.  The  two-stage  scheme 
yielded  an  AUC  of  0.83  while  the  one-stage  scheme  yielded 
an  AUC  of  0.67,  with  difference  being  statistically  significant 
(p<10“^).  The  inferior  performance  of  the  one- stage  scheme 


can  be  mainly  explained  by  the  fact  that  a  single  BANN 
classifier  lacks  the  ability  to  deal  with  features  in  a  pairwise 
way,  thus  the  information  regarding  correlation  between  fea¬ 
ture  pairs  cannot  be  efficiently  utilized. 

in  order  to  evaluate  how  the  pathology  of  lesions  affects 
the  performance  of  the  proposed  method,  the  entire  dataset 
was  split  into  benign  and  malignant  subsets,  as  described  in 
Table  I.  As  noted  earlier,  the  correlation  value  between  pair¬ 
wise  features,  and  not  the  feature  value  itself,  plays  a  crucial 
role  in  the  task  of  distinguishing  corresponding  and  noncor¬ 
responding  image  pairs,  thus  we  compared  the  correlation 
coefficients  between  image  pairs  for  benign  and  malignant 
lesions,  respectively.  We  failed  to  observe  significant  differ¬ 
ence  for  most  of  features  between  benign  and  malignant  le¬ 
sions,  as  shown  in  Fig.  9.  The  results  indicate  that  the  pair¬ 
wise  feature  analysis  may  be  independent  of  pathology. 

Due  to  the  database  size,  there  are  two  limitations  in  this 
preliminary  study.  First,  the  proposed  correlative  feature 
analysis  was  only  applied  on  CC  versus  M  L  views,  however, 
pairing  other  views,  such  as  CC  versus  M  LO  and  M  L  versus 
MLO,  is  also  commonly  used  in  clinical  practice.  Thus,  in 
further  study,  we  will  evaluate  the  computerized  analysis  on 
those  view  pairs  and  investigate  how  the  different  pairwise 
views  affect  the  performance  of  proposed  analysis.  Second, 
for  noncorresponding  pairs,  lesions  could  be  with  either 
same  pathology  (i.e.,  both  malignant  or  both  benign)  or  dif¬ 
ferent  pathology  (i.e.,  one  malignant  and  one  benign).  Spe¬ 
cifically,  we  are  more  interested  in  noncorresponding  lesions 
of  different  pathology  since  integrating  information  from  le¬ 
sions  with  different  pathology  would  hinder  more  the  perfor¬ 
mance  of  CAD  systems.  However,  we  regarded  the  noncor¬ 
responding  lesion  pairs  as  a  whole  in  this  study  as  there  are 
only  28  image  pairs  with  different  pathology.  The  perfor¬ 
mance  of  the  proposed  analysis  on  noncorresponding  lesion 
pairs  with  different  pathology,  and  the  mismatching  effects 
on  the  CAD  performance  are  interesting  research  questions 
for  our  future  study. 

V.  CONCLUSION 

in  this  paper,  we  have  presented  a  novel  two-BANN  cor¬ 
relative  feature  analysis  framework  to  estimate  the  probabil¬ 
ity  that  a  given  pair  of  two  images  is  of  the  same  physical 
lesion.  Our  investigation  indicates  that  the  proposed  method 
is  a  promising  way  to  distinguish  between  corresponding  and 
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Fig.  9.  Correlation  coefficients  between  CC  and  M  L  views  and  associated  95%  confidence  intervals  of  the  18  features  extracted  from  benign  (solid)  and 
malignant  (dash)  lesions.  All  the  lesions  were  segmented  via  an  automatic  segmentation  algorithm.  Left:  corresponding  image  pairs.  Right:  noncorresponding 
image  pairs. 


noncorresponding  pairs.  With  ieave-one-out  (by  iesion)  cross 
vaiidation,  the  distance-feature-based  correspondence  metric 
yieided  an  AUC  of  0.81  and  a  feature  correspondence  metric 
subset,  which  inciudes  distance,  gradient  texture,  and  ROl- 
based  correiation,  yieided  an  AUC  of  0.87.  The  improve¬ 
ment  by  using  muitipie  feature  correspondence  metrics  was 
statisticaiiy  significant  compared  to  singie  feature  metric  per¬ 
formance.  This  method  has  the  potentiai  to  be  generaiized 
and  empioyed  to  differentiating  corresponding  and  noncorre¬ 
sponding  pairs  from  muiti-modaiity  breast  imaging. 
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PURPOSE: 

The  objective  of  this  study  is  to  evaluate  the  correlation  performance  of  individual 
computerized  features  extracted  from  the  full  field  digital  mammograms  (FFDM)  of  a 
lesion  obtained  in  two  different  views.  This  research  provides  a  guide  for  discriminating 
corresponding  and  non-corresponding  lesion  pairs  within  the  CAD  framework.  It  is  also 
helpful  for  guiding  the  development  of  new  features  to  improve  the  accuracy  of  image 
matching  in  disease  diagnosis  and  prognosis. 


METHOD  AND  MATERIALS: 

One  dataset  (A)  includes  103  biopsy  proven  cases  (48  benign  solid  lesions  and  55 
malignant  lesions),  each  of  which  has  a  craniocaudal  (CC)  and  mediolateral  (ML)  view. 
Another  dataset  (B)  includes  52  cases  (24  benign  solid  lesions  and  28  malignant  lesions), 
each  of  which  has  a  CC  and  mediolateral  oblique  (MLO)  view.  In  order  to  evaluate  the 
robustness  of  the  correlation  performance  to  lesion  segmentation,  besides  the 
radiologist’s  outlines,  three  automatic  segmentation  methods  were  employed  to  extract 
the  mass  lesion  from  the  surrounding  tissues.  The  conventional  region-growing  method 
uses  abrupt  changes  in  size  and  circularity  as  the  rules  of  segmentation.  The  radial 
gradient  index  (RGI)  based  method  applies  RGI  model  to  the  suspicious  lesion  multiplied 
by  a  constraint  function.  The  region-based  active  contour  model  evolves  the  contour 
based  on  the  homogeneities  both  inside  and  outside  of  the  evolving  contour.  Fifteen 
computer-extracted  features  of  each  lesion  were  calculated  in  both  views  in  order  to 
quantify  the  characteristics  of  margin,  shape,  contrast  and  texture  of  the  lesion.  For  each 
feature,  correlation  coefficient  between  the  two  views  and  the  p-value  of  the  derived 
correlation  coefficient  were  obtained. 


RESULTS: 

With  the  human  outline,  the  feature  characterizing  the  diameter  of  lesion  yielded  the 
correlation  efficient  of  0.87  for  dataset  A  and  0.88  for  dataset  B,  both  of  which  have  p- 
values  far  less  than  0.05.  The  features  characterizing  shape,  contrast  and  texture  showed 
better  performance  among  the  1 5  individual  features  despite  of  segmentation  methods, 
pathology  and  the  type  of  view  pairs.  This  is  because  the  features  representing  large-scale 
information  are  less  sensitive  to  the  change  of  position  than  those  representing  small- 


scale  information,  which  results  in  the  higher  correlation  between  large-scale  features 
from  different  views  than  that  of  small-scale  features. 

CONCLUSIONS: 

Our  investigation  indicates  that  the  features  that  characterize  the  large-scale  information 
of  lesion  have  higher  correlation  between  the  two  view  images.  We  are  currently 
applying  these  features  to  develop  automated  image  matching  method  to  determine 
corresponding  and  non-corresponding  lesion  pairs. 
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Abstract.  In  this  study,  we  present  a  computerized  framework  to  iden¬ 
tify  the  corresponding  image  pair  of  a  lesion  in  CC  and  MLO  views,  a 
prerequisite  for  combining  information  from  these  views  to  improve  the 
diagnostic  ability  of  both  radiologists  and  CAD  systems.  A  database  of 
126  mass  lesons  was  used,  from  which  a  corresponding  dataset  with  104 
pairs  and  a  non-corresponding  dataset  with  95  pairs  were  constructed. 
For  each  FFDM  image,  the  mass  lesions  were  firstly  automatically  seg¬ 
mented  via  a  dual-stage  algorithm,  in  which  a  RGI-based  segmentation 
and  an  active  contour  model  are  employed  sequentially.  Then,  various 
features  were  automatically  extracted  from  the  lesion  to  characterize  the 
spiculation,  margin,  size,  texture  and  context  of  the  lesion,  as  well  as  its 
distance  to  nipple.  We  developed  a  two-step  strategy  to  select  an  effective 
subset  of  features,  and  combined  it  with  a  BANN  to  estimate  the  proba¬ 
bility  that  the  two  images  are  of  the  same  physical  lesion.  ROC  analysis 
was  used  to  evaluate  the  performance  of  the  individual  features  and  the 
selected  feature  subset  for  the  task  of  distinguishing  corresponding  and 
non-corresponding  pairs.  With  leave-one-out  evaluation  by  lesion,  the 
distance  feature  yielded  an  AUC  of  0.78  and  the  feature  subset,  which  in¬ 
cludes  distance,  ROI-based  energy  and  ROI-based  homogeneity,  yielded 
an  AUC  of  0.88.  The  improvement  by  using  multiple  features  was  statis¬ 
tically  significant  compared  to  single  feature  performance  {p  <  0.001) 


1  Introduction 


In  mammographic  screening,  different  projections  provide  complementary  infor¬ 
mation  about  the  same  physical  lesion,  and  thus,  it  has  been  well  recognized  that 
multiple  views  can  improve  the  diagnosis  of  breast  cancer  in  the  computerized 
analysis  of  mammograms  [1-3].  To  merge  information  from  images  of  different 
views,  an  essential  step  is  to  clarify  if  these  images  actually  represent  the  same 
physical  lesion,  as  Fig.  1  shows.  In  this  study,  we  present  a  correlative  feature 
analysis  (CFA)  framework  to  address  the  task  of  identifying  corresponding  im¬ 
ages  of  lesions  as  seen  with  craniocaudal  (CC)  and  mediolateral  oblique  (MLO) 
views. 


Fig.  1.  An  example  of  two  lesions  in  the  same  breast  seen  in  CC  view  (left)  and  MLO 
view  (right).  The  arrow  indicates  the  correspondence  of  the  same  physical  lesion  in 
different  views. 


2  Database 

The  full-field  digital  mammography  (FFDM)  database  consists  of  126  biopsied 
lesions  obtained  from  GE  Senographe  2000D  systems  with  spatial  resolution  of 
O.lmm  X  0.1mm.  The  mass  lesions  were  identified  and  outlined  by  an  expert 
breast  radiologist  based  on  visual  criterion  and  biopy-proven  reports.  The  distri¬ 
bution  of  lesion  size,  which  is  characterized  as  the  equivalent  diameter  of  the  area 
inside  the  radiologist's  manually  delineated  contour,  is  shown  in  Fig.  2.  Based 
on  the  correspondence  of  lesions  identified  by  the  radiologist,  we  constructed 
104  corresponding  and  95  non-corresponding  image  pairs.  Each  pair  consists  of 
a  CC  view  and  a  MLO  view.  Considering  the  most  realistic  scenario  of  lesion 
mismatch  in  clinical  practice,  the  non-corresponding  pairs  were  built  from  cases 
of  the  same  patients  but  different  physical  lesions. 

3  Methods 

3.1  Lesion  segmentation 

A  dual-stage  segmentation  method  was  initially  applied  to  extract  mass  lesions 
from  the  surrounding  tissues  [4].  This  algorithm  utilizes  an  active  contour  model 
that  maximizes  a  cost  function  based  on  the  homogeneities  inside  and  outside 
of  the  evolving  contour  [5].  Prior  to  the  application  of  the  active  contour  model, 
a  radial  gradient  index  (RGI)  based  algorithm  [6]  is  employed  to  yield  an  ini¬ 
tial  contour  close  to  the  lesion  boundary  location  in  a  computationally  efficient 
manner.  Based  on  the  initial  segmentation,  an  automatic  background  estimation 
method  is  applied  to  identify  the  effective  circumstance  of  the  lesion.  In  addi¬ 
tion,  instead  of  empirically-determined  criteria  such  as  fixed  iteration  times,  a 
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Fig.  2.  Distribution  of  lesions'  equivalent  diameters  obtained  from  the  FFDM  database. 


dynamic  stopping  criterion  is  implemented  to  terminate  the  contour  evolution 
when  it  reaches  the  lesion  boundary. 


3.2  Feature  extraction 

In  our  study  the  computer-extracted  features  were  grouped  into  three  categories. 
The  first  category  includes  features  quantifying  margin  sharpness,  spiculation, 
gradient,  contrast  and  shape  of  a  lesion  [7].  The  second  category  includes  tex¬ 
ture  features  extracted  from  two  regions,  i.e.  the  lesion  and  the  entire  encom¬ 
passing  ROI,  respectively.  For  each  region,  a  2D  gray-level  co-occurrence  matrix 
(GLCM)  was  constructed,  and  texture  features  were  extracted  to  characterize 
homogeneity,  gray-level  dependence,  brightness,  variation  and  randomness  [8]. 
We  developed  an  automatic  neighborhood  estimation  method  to  determine  the 
effective  circumstance  of  the  lesion. 

The  third  group  includes  a  distance  feature  calculated  as  the  Euclidean  dis¬ 
tance  from  the  nipple  location  to  the  center  of  the  lesion.  Since  nipple  markers, 
which  present  as  bright  markers  on  the  mammograms,  are  commonly  used  in 
mamographic  images,  we  developed  a  nipple  identification  method  by  locating 
those  markers  automatically.  This  method  includes  several  processing  steps,  as 
illustrated  in  Fig.  3.  Initially,  gray-level  thresholding  is  employed  on  the  en¬ 
tire  mammogram  to  extract  the  breast  region  from  the  external-to-breast  back¬ 
ground.  Then,  another  gray-level  threshold  is  applied  to  the  breast  region,  yield¬ 
ing  several  nipple  marker  candidates.  The  breast  skin  boundary  is  obtained  by 
subtracting  a  morphologically  eroded  [9]  breast  region  from  its  original  region. 


To  reduce  the  number  of  falsely  identified  nipple  markers,  area  and  circular¬ 
ity  constraints  are  imposed  on  each  candidates,  and  those  candidates  with  area 
within  a  given  range  and  circularity  above  a  certain  threshold  are  kept  for  the 
final  step.  The  area  range  and  circularity  threshold  were  empirically  determined 
with  10  randomly  selected  images  in  this  study.  The  nipple  marker  is  finally 
chosen  as  the  one  closest  to  the  breast  boundary. 


Fig.  3.  Schematic  diagram  of  the  proposed  automatic  nipple  identification  algorithm. 


3.3  Feature  selection  and  classification 

For  each  pair-wise  set  of  features  in  representing  the  two  different  views,  a 
Bayesian  artificial  neural  network  (BANN)  classifier  [10]  was  employed  to  merge 
each  feature  pair  into  a  correspondence  metric,  which  estimates  the  probabil¬ 
ity  that  the  two  images  are  of  the  same  physical  lesion,  based  on  that  specific 
feature.  Next,  an  effective  subset  of  correspondence  metrics  was  selected  via  a 
linear  stepwise  feature  selection  [11]  with  a  Wilks  lambda  criterion,  and  merged 
with  another  BANN  to  yield  an  overall  estimateion  of  the  probability  of  corre¬ 
spondence. 

3.4  Evaluation 

The  area  under  the  receiver  operating  characteristic  (ROC)  curve  (AUC)  [12][13] 
was  used  as  an  index  of  performance  of  the  individual  features  and  the  classi¬ 
fier  outputs  in  the  task  of  distinguishing  between  corresponding  pairs  from  and 


non-corresponding  ones.  The  levels  of  statistical  significance  among  individual 
features,  and  single  feature  versus  merged  multiple  features,  were  calculated  by 
ROCKIT  software  (version  1.1&). 

4  Results 

In  a  leave-one-out  evaluation  by  lesion,  the  distance  feature  outperformed  among 
all  the  other  individual  features,  yielding  an  AUC  of  0.78.  The  selected  feature 
subset,  which  includes  distance,  ROTbased  energy  and  ROI-based  homogeneity, 
yielded  an  AUC  of  0.88.  The  improvement  by  using  multiple  features  was  sta¬ 
tistically  significant  compared  to  single  feature  performance  (p  <  0.001).  ROC 
curves  resulting  from  evaluation  of  the  distance  feature  and  the  feature  subset 
are  shown  in  Fig.  4. 


Fig.  4.  ROC  curves  of  leave-one-out  evaluation  by  lesion  for  distance  feature  and  the 
feature  subset  (distance  feature,  ROI-based  energy  and  ROI-based  homogeneity) . 


5  Discussion 

We  presented  here  a  correlative  feature  analysis  framework  to  assess  the  proba¬ 
bility  that  a  given  pair  of  two  mammographic  images  from  CC  and  MLO  view 


is  of  the  same  physical  lesion.  Our  results  are  promising  into  distinguishing  be¬ 
tween  corresponding  and  non-corresponding  lesion  pairs.  It  is  important  to  note 
that  our  method  is  feature-based,  which  employs  a  BANN  classifier  to  estimate 
the  relationship  between  computer-extracted  features  of  a  lesion  in  CC  and  MLO 
views.  The  supervised- learning  manner  cannot  only  make  the  relationship  flex¬ 
ible  to  each  feature,  but  also  avoid  the  sophisticated  geometrically-deformable 
models  that  are  widely  used  in  geometric  breast  registrations. 

We  are  generalizing  this  framework  and  applying  it  to  differentiate  corre¬ 
sponding  and  non-corresponding  pairs  from  multi-modality  breast  images,  such 
as  FFDM  and  breast  MRI  images. 
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ABSTRACT 

Since  different  imaging  modalities  provide  complementary  information  regarding  the  same  lesion,  combining  in¬ 
formation  from  different  modalities  may  increase  diagnostic  accuracy.  In  this  study,  we  investigated  the  use  of 
computerized  features  of  lesions  imaged  via  both  full-field  digital  mammography  (FFDM)  and  dynamic  contrast- 
enhanced  magnetic  resonance  imaging  (DCE-MRI)  in  the  classification  of  breast  lesions.  Using  a  manually 
identified  lesion  location,  i.e.  a  seed  point  on  FFDM  images  or  a  ROI  on  DCE-MRI  images,  the  computer  auto¬ 
matically  segmented  mass  lesions  and  extracted  a  set  of  features  for  each  lesion.  Linear  stepwise  feature  selection 
was  firstly  performed  on  single  modality,  yielding  one  feature  subset  for  each  modality.  Then,  these  selected 
features  served  as  the  input  to  another  feature  selection  procedure  when  extracting  useful  information  from  both 
modalities.  The  selected  features  were  merged  by  linear  discriminant  analysis  (LDA)  into  a  discriminant  score. 
Receiver  operating  characteristic  (ROC)  analysis  was  used  to  evaluate  the  performance  of  the  selected  feature 
subset  in  the  task  of  distinguishing  between  malignant  and  benign  lesions.  From  a  FFDM  database  with  321 
lesions  (167  malignant  and  154  benign),  and  a  DCE-MRI  database  including  181  lesions  (97  malignant  and  84 
benign),  we  constructed  a  multi-modality  dataset  with  51  lesions  (29  malignant  and  22  benign).  With  leave-one- 
out-by- lesion  evaluation  on  the  multi-modality  dataset,  the  mammography-only  features  yielded  an  area  under 
the  ROC  curve  (AUC)  of  0.62  ±  0.08  and  the  DCE-MRI-only  features  yielded  an  AUC  of  0.90  ±  0.05.  The  com¬ 
bination  of  these  two  modalities,  which  included  a  spiculation  feature  from  mammography  and  a  kinetic  feature 
from  DCE-MRI,  yielded  an  AUC  of  0.94  ±0.03.  The  improvement  of  combining  multi-modality  information  was 
statistically  significant  as  compared  to  the  use  of  mammography  only  {p  =  0.0001).  However,  we  failed  to  show 
the  statistically  significant  improvement  as  compared  to  DCE-MRI,  with  the  limited  multi-modality  dataset 

(p  =  0.22). 

Keywords:  Breast  cancer,  mammography,  DCE-MRI,  computer-aided  diagnosis 

1.  INTRODUCTION 

Breast  cancer  is  the  most  common  malignancy  in  American  women  and  the  second  most  common  cause  of  deaths 
from  malignancy  in  the  population  (after  lung  cancer).  According  to  the  American  Cancer  Society,  about  182, 460 
women  in  the  United  States  will  be  found  to  have  invasive  breast  cancer  in  2008,  and  about  40,480  women  will 
die  from  the  disease  this  year.^  Although  there  are  currently  limited  methods  for  curing  breast  cancer,  early 
detection  by  breast  imaging  plays  an  important  role  in  reducing  the  mortality.  Between  the  years  1991  and  2003, 
there  has  been  a  steady  decrease  in  the  annual  death  rate  from  female  breast  cancer,  from  32.69  to  25.19  (per 
100,000  population).  This  decrease  largely  reflects  improvements  in  early  detection  and  treatment. 

Although  mammography  has  achieved  significant  success  and  reduces  the  mortality  from  breast  cancer  by 
15%  —  35%,^’^  it  is  far  from  perfect:  about  15%  —  20%  of  cancers  are  missed,  and  75%  of  lesions  detected  by 
mammography  are  benign,  resulting  in  many  unnecessary  medical  procedures,  including  biopsies.^  Consequently, 
some  complementary  imaging  modalities,  such  as  breast  DCE-MRI  and  breast  sonography,  are  being  investigated 
to  improve  the  sensitivity  and  specificity  of  breast  cancer  detection  and  diagnosis. 

While  the  results  of  computer-aided  diagnosis  (CAD)  systems  for  single  imaging  modality  are  encouraging, 
merging  information  across  different  modalities  is  recently  attracting  more  attention.  Because  different  imag¬ 
ing  modalities  provide  complementary  information  regarding  lesions,  combining  information  from  two  or  more 
modalities  may  increase  the  diagnostic  accuracy.  Several  investigations  have  been  conducted  to  combine  informa¬ 
tion  from  mammography  and  sonography  to  improve  the  diagnosis  of  breast  cancer.®^®  Their  results  showed  that 
the  performance  of  cancer  classification  was  improved  as  compared  to  each  individual  modality.  However,  to  the 


Figure  1.  Example:  a  malignant  lesion  imaged  by  both  mammography  (left)  and  DCE-MRI  (right).  The  solid  lines  in  both 
images  are  the  segmentation  results  from  the  dual-stage  method®  and  the  fuzzy  c-means  based  method/®  respectively. 


best  of  our  knowledge,  there  is  limited  research  on  combining  information  from  mammography  and  DCE-MRI. 
Thus,  in  this  study,  we  investigated  the  use  of  computerized  features  of  lesions  imaged  via  both  mammography 
and  DCE-MRI  (Fig.  1)  in  the  classification  of  breast  lesions. 

2.  METHODS 

Our  computerized  scheme  consists  of  several  steps:  1)  automatic  extraction  of  lesions  on  each  modality  images  via 
computerized  segmentation  methods;  2)  automatic  extraction  of  various  features  (mathematical  descriptors)  from 
lesions;  and  3)  merging  of  information  from  different  modalities  and  to  estimate  the  probability  of  malignancy. 

2.1.  Lesion  segmentation 

For  FFDM  images,  a  dual-stage  method  was  employed  to  automatically  extract  lesions  from  the  normal  breast 
tissue.®  In  this  method,  a  radial  gradient  index  (RGI)  based  segmentation^^  is  used  to  yield  an  initial  contour 
close  to  the  lesion  boundary  location  in  a  computationally  efficient  manner.  This  initial  segmentation  also  pro¬ 
vides  a  base  to  identify  the  effective  circumstance  of  the  lesion  via  an  automatic  background  estimation  method. 
Then  a  region-based  active  contour  mode/®  is  utilized  to  evolve  the  contour  further  to  the  lesion  boundary.  The 
active  contour  model  relies  on  an  intrinsic  property  of  image  segmentation  that  each  segmented  region  should  be 
as  homogeneous  as  possible  for  an  image  formed  by  two  regions.  Instead  of  empirically  determined  criteria  such 
as  fixed  iteration  times,  a  dynamic  stopping  criterion  is  implemented  to  terminate  the  contour  evolution  when 
it  reaches  the  lesion  boundary. 

For  DCE-MRI  images,  a  fuzzy  c-means  (FCM)  clustering-based  method  was  used  for  the  segmentation  of 
lesions  in  3D  space. This  scheme  includes  six  steps.  An  ROI  is  first  selected  by  a  human  operator;  then  the 
post-contrast  ROI  series  are  enhanced  by  dividing  the  pixel  value  at  each  voxel  by  the  value  at  the  corresponding 
pre-contrast  voxel.  After  the  FCM  clustering  method  is  applied  to  partition  the  whole  ROI  into  lesion  and 
non-lesion  parts,  the  lesion  membership  map  is  binarized  with  an  empirically  determined  threshold.  Then  a  3D 
connected-component  labeling  operation  is  performed  to  reduce  the  false-positive  voxels.  Finally,  a  hole-filling 
operation  is  performed  yielding  the  final  segmented  lesion. 

2.2.  Computerized  feature  extraction 

For  FFDM  images,  fifteen  features  were  extracted  from  the  segmented  lesions,  which  characterize  spiculation, 
margin  sharpness,  shape  and  density  of  the  lesions. In  our  FFDM  database,  the  number  of  mammograms 
available  for  each  physical  lesion  was  different,  ranging  from  1  to  13.  Thus,  for  each  feature,  we  determined  a 
representative  value  of  a  lesion  as  its  average  value  over  all  the  mammograms  of  that  particular  lesion. 

For  DCE-MRI  images,  another  15  features  were  extracted  from  the  lesions  in  3D  space,  which  included 
spiculation  features,  shape  features,  enhancement-kinetics-based  features  and  enhancement-variance  dynamics 


Figure  2.  Flow  chart  of  feature  selection  and  classification.  . 

features.^"*  The  characteristic  kinetic  curves  were  generated  from  the  “most-enhanced”  regions,  which  were 
automatically  identified  by  a  FCM  clustering  method.^® 

2.3.  Feature  selection  and  classification 

In  this  study,  the  feature  selection  procedure  included  two  steps,  as  shown  in  Fig.  2.  Linear  stepwise  feature 
selection,^®  with  Wilke’s  lambda  criterion,  was  firstly  performed  on  single  modalities,  yielding  one  feature  subset 
for  each  modality.  Then,  these  selected  features  served  as  the  input  to  another  feature  selection  procedure  when 
extracting  useful  information  from  both  modalities. 

Linear  discriminant  analysis  (LDA)^^  was  employed  to  merge  the  selected  features  to  a  single  discriminant 
score  that  is  related  to  the  estimated  likelihood  of  malignancy. 

2.4.  Performance  evaluation  and  statistical  analysis 

The  performance  of  discriminant  scores  from  both  single  modality  and  multiple  modalities,  in  the  task  of  dif¬ 
ferentiating  malignant  lesions  from  benign  ones,  was  evaluated  using  receiver  operating  characteristic  (ROC) 
analysis, ^he  area  under  the  ROC  curve  (AUC)  as  a  figure  of  merit.  The  level  of  statistical  significance 
was  calculated  by  ROCKIT  software  (version  1.1b).  A  leave-one-out-by-lesion  evaluation  was  used  to  evaluate 
the  performance  of  each  classifier. 

3.  RESULTS 

In  this  preliminary  study,  we  used  a  FFDM  database  including  321  lesions  (167  malignant  and  154  benign),  and 
a  DCE-MRI  database  including  181  lesions  (97  malignant  and  84  benign).  All  the  lesions  are  biopsy-proven. 
From  these  two  databases,  we  constructed  a  multi-modality  dataset  of  51  lesions  (29  malignant  and  22  benign). 
Mammograms  and  DCE-MRI  images  are  available  for  these  lesions. 

With  the  entire  FFDM  database,  ROI-based  normalized  radial  gradient  (NRG),  lesion-based  NRG,  and 
gradient  texture  were  selected  as  an  effective  feature  subset.  The  leave-one-out-by- lesion  evaluation  using  LDA 
to  merge  the  selected  features  yielded  an  AUC  value  of  0.62  ±  0.08  on  the  multi-modality  dataset. 


Spiculation  from  mammography 


Figure  3.  Scatter  plot  of  the  spiculation  feature  from  mammography  versus  the  peak  location  of  the  kinetic  curve  from 
DCE-MRI. 

With  the  entire  DCE-MRI  database,  margin  sharpness,  irregularity,  peak  location  of  the  enhancement  kinet¬ 
ics,  and  enhancement- variance  uptake  rate  were  selected  as  an  effective  feature  subset.  The  leave-one-out-by- 
lesion  evaluation  using  LDA  yielded  an  AUC  value  of  0.90  ±  0.05  on  the  multi-modality  dataset. 

With  the  multi-modality  dataset,  the  lesion-based  NRG  from  mammography  and  the  peak  location  of  the 
enhancement  kinetics  from  DCE-MRI  were  selected  from  the  above  7  features.  Figure  3  shows  the  scatter  plot 
of  these  two  features  on  the  multi-modality  dataset.  The  leave-one-out-by-lesion  evaluation  using  LDA  yielded 
an  AUC  of  0.94  ±0.03. 

The  improvement  of  combining  multi-modality  information  was  statistically  significant  as  compared  to  the 
use  of  mammography  only  {p  =  0.0001).  However,  although  the  performance  of  combining  multi-modality 
information  was  better  than  the  use  of  DCE-MRI  only,  we  failed  to  show  the  statistically  significant  improvement, 
with  the  limited  multi-modality  dataset  {p  =  0.22). 

4.  CONCLUSION 

In  the  proposed  study,  we  investigated  the  performance  of  a  computerized  classification  scheme  with  computer- 
extracted  features  based  on  mammography  alone,  DCE-MRI  alone,  and  the  combination  of  these  two  modalities. 
In  mammography,  spiculation  and  texture  features  were  shown  to  be  effective  for  breast  cancer  classification. 
In  DCE-MRI  images,  margin  sharpness,  lesion  shape  and  kinetic  features  were  salient.  In  our  previous  studies, 
spiculation  and  kinetic  features  have  been  justified  as  the  best  features  when  distinguishing  malignant  and  benign 
lesions  for  mammography  and  DCE-MRI,  respectively.  Our  feature  selection  method  correctly  captured  these 
two  features  when  combining  information  across  different  modalities. 

Our  pilot  results  showed  that  combining  information  from  multiple  modalities  performed  better  than  the 
single  modality  in  the  task  of  distinguishing  between  malignant  and  benign  lesions.  We  are  currently  expanding 
our  multi-modality  database  and  will  evaluate  the  performance  of  combining  information  from  multi-modalities 
on  the  larger  database. 
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